2024-05-08T15:42:26.759 INFO:root:teuthology version: 0.0.1.dev271+g1ae7ad8 2024-05-08T15:42:26.760 DEBUG:teuthology.run:Teuthology command: teuthology --description rados/standalone/{supported-random-distro$/{ubuntu_latest} workloads/erasure-code} --owner scheduled_lflores@teuthology --archive /home/teuthworker/archive/lflores-2024-05-08_14:59:36-rados-wip-lflores-testing-2-2024-05-07-1606-squid-distro-default-smithi/7697621 --name lflores-2024-05-08_14:59:36-rados-wip-lflores-testing-2-2024-05-07-1606-squid-distro-default-smithi --verbose -- /home/teuthworker/archive/lflores-2024-05-08_14:59:36-rados-wip-lflores-testing-2-2024-05-07-1606-squid-distro-default-smithi/7697621/orig.config.yaml 2024-05-08T15:42:26.786 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-05-08T15:42:26.864 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/lflores-2024-05-08_14:59:36-rados-wip-lflores-testing-2-2024-05-07-1606-squid-distro-default-smithi/7697621 branch: wip-lflores-testing-2-2024-05-07-1606-squid description: rados/standalone/{supported-random-distro$/{ubuntu_latest} workloads/erasure-code} email: null first_in_suite: false job_id: '7697621' kernel: kdb: 1 sha1: distro ktype: distro last_in_suite: false machine_type: smithi name: lflores-2024-05-08_14:59:36-rados-wip-lflores-testing-2-2024-05-07-1606-squid-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 3 size: 10 os_type: ubuntu os_version: '22.04' overrides: admin_socket: branch: wip-lflores-testing-2-2024-05-07-1606-squid ceph: conf: mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug ms: 1 debug osd: 20 flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: 95a4ad3c90f2f12f920e87725ad8d9002f948722 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: 95a4ad3c90f2f12f920e87725ad8d9002f948722 workunit: branch: wip-lflores-testing-2-2024-05-07-1606-squid sha1: 95a4ad3c90f2f12f920e87725ad8d9002f948722 owner: scheduled_lflores@teuthology priority: 75 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.x - osd.0 - osd.1 - osd.2 - client.0 seed: 8082 sha1: 95a4ad3c90f2f12f920e87725ad8d9002f948722 sleep_before_teardown: 0 subset: 111/120000 suite: rados suite_branch: wip-lflores-testing-2-2024-05-07-1606-squid suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_95a4ad3c90f2f12f920e87725ad8d9002f948722/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 95a4ad3c90f2f12f920e87725ad8d9002f948722 targets: smithi172.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJevBFpKZ74PzLqEbvStXF0lubWr5uthtHl/AfagUmfj7lmRa4iGrY4hz6c1lyM4N0HCBnw5BuOvjTbIAkT0Dyo= tasks: - install: null - workunit: basedir: qa/standalone clients: all: - erasure-code teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 1ae7ad82388e92a475afff437d49054826c019a1 timestamp: 2024-05-08_14:59:36 tube: smithi user: lflores verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-05-08T15:42:26.864 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_95a4ad3c90f2f12f920e87725ad8d9002f948722/qa; will attempt to use it 2024-05-08T15:42:26.865 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_95a4ad3c90f2f12f920e87725ad8d9002f948722/qa/tasks 2024-05-08T15:42:26.865 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-05-08T15:42:26.867 INFO:teuthology.task.internal:Checking packages... 2024-05-08T15:42:26.892 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'default' and ceph hash '95a4ad3c90f2f12f920e87725ad8d9002f948722' 2024-05-08T15:42:26.892 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-05-08T15:42:26.892 INFO:teuthology.packaging:ref: None 2024-05-08T15:42:26.892 INFO:teuthology.packaging:tag: None 2024-05-08T15:42:26.892 INFO:teuthology.packaging:branch: wip-lflores-testing-2-2024-05-07-1606-squid 2024-05-08T15:42:26.892 INFO:teuthology.packaging:sha1: 95a4ad3c90f2f12f920e87725ad8d9002f948722 2024-05-08T15:42:26.892 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&ref=wip-lflores-testing-2-2024-05-07-1606-squid 2024-05-08T15:42:27.111 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-2517-g95a4ad3c-1jammy 2024-05-08T15:42:27.113 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-05-08T15:42:27.119 INFO:teuthology.task.internal:no buildpackages task found 2024-05-08T15:42:27.119 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-05-08T15:42:27.142 INFO:teuthology.task.internal:Saving configuration 2024-05-08T15:42:27.155 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-05-08T15:42:27.166 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-05-08T15:42:27.193 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi172.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/lflores-2024-05-08_14:59:36-rados-wip-lflores-testing-2-2024-05-07-1606-squid-distro-default-smithi/7697621', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-05-08 15:36:30.472480', 'locked_by': 'scheduled_lflores@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIG+Q7qrpXoBaMj4eY5Y80O85jzAZG6mHgt6VI1qD0SRe'} 2024-05-08T15:42:27.193 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-05-08T15:42:27.199 INFO:teuthology.task.internal:roles: ubuntu@smithi172.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0'] 2024-05-08T15:42:27.199 INFO:teuthology.run_tasks:Running task console_log... 2024-05-08T15:42:27.256 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f2e4ad9df70>, signals=[15]) 2024-05-08T15:42:27.256 INFO:teuthology.run_tasks:Running task internal.connect... 2024-05-08T15:42:27.262 INFO:teuthology.task.internal:Opening connections... 2024-05-08T15:42:27.262 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi172.front.sepia.ceph.com 2024-05-08T15:42:27.264 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi172.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-08T15:42:27.349 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-05-08T15:42:27.356 DEBUG:teuthology.orchestra.run.smithi172:> uname -m 2024-05-08T15:42:27.462 INFO:teuthology.orchestra.run.smithi172.stdout:x86_64 2024-05-08T15:42:27.463 DEBUG:teuthology.orchestra.run.smithi172:> cat /etc/os-release 2024-05-08T15:42:27.512 INFO:teuthology.orchestra.run.smithi172.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS" 2024-05-08T15:42:27.512 INFO:teuthology.orchestra.run.smithi172.stdout:NAME="Ubuntu" 2024-05-08T15:42:27.512 INFO:teuthology.orchestra.run.smithi172.stdout:VERSION_ID="22.04" 2024-05-08T15:42:27.513 INFO:teuthology.orchestra.run.smithi172.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)" 2024-05-08T15:42:27.513 INFO:teuthology.orchestra.run.smithi172.stdout:VERSION_CODENAME=jammy 2024-05-08T15:42:27.513 INFO:teuthology.orchestra.run.smithi172.stdout:ID=ubuntu 2024-05-08T15:42:27.513 INFO:teuthology.orchestra.run.smithi172.stdout:ID_LIKE=debian 2024-05-08T15:42:27.513 INFO:teuthology.orchestra.run.smithi172.stdout:HOME_URL="https://www.ubuntu.com/" 2024-05-08T15:42:27.513 INFO:teuthology.orchestra.run.smithi172.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2024-05-08T15:42:27.513 INFO:teuthology.orchestra.run.smithi172.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2024-05-08T15:42:27.513 INFO:teuthology.orchestra.run.smithi172.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2024-05-08T15:42:27.513 INFO:teuthology.orchestra.run.smithi172.stdout:UBUNTU_CODENAME=jammy 2024-05-08T15:42:27.514 INFO:teuthology.lock.ops:Updating smithi172.front.sepia.ceph.com on lock server 2024-05-08T15:42:27.545 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-05-08T15:42:27.553 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-05-08T15:42:27.566 INFO:teuthology.task.internal:Checking for old test directory... 2024-05-08T15:42:27.567 DEBUG:teuthology.orchestra.run.smithi172:> test '!' -e /home/ubuntu/cephtest 2024-05-08T15:42:27.571 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-05-08T15:42:27.592 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-05-08T15:42:27.592 DEBUG:teuthology.orchestra.run.smithi172:> test -z $(ls -A /var/lib/ceph) 2024-05-08T15:42:27.621 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-05-08T15:42:27.661 INFO:teuthology.run_tasks:Running task kernel... 2024-05-08T15:42:27.679 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-05-08T15:42:27.679 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-05-08T15:42:27.680 DEBUG:teuthology.orchestra.run.smithi172:> test -f /run/.containerenv -o -f /.dockerenv 2024-05-08T15:42:27.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-08T15:42:27.685 DEBUG:teuthology.orchestra.run.smithi172:> uname -r 2024-05-08T15:42:27.733 INFO:teuthology.orchestra.run.smithi172.stdout:5.15.0-56-generic 2024-05-08T15:42:27.733 INFO:teuthology.task.kernel:Running kernel on smithi172: 5.15.0-56-generic 2024-05-08T15:42:27.733 DEBUG:teuthology.orchestra.run.smithi172:> sudo apt-get clean 2024-05-08T15:42:27.871 DEBUG:teuthology.orchestra.run.smithi172:> sudo apt-get update 2024-05-08T15:42:28.020 INFO:teuthology.orchestra.run.smithi172.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-05-08T15:42:28.044 INFO:teuthology.orchestra.run.smithi172.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [119 kB] 2024-05-08T15:42:28.113 INFO:teuthology.orchestra.run.smithi172.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-backports InRelease [109 kB] 2024-05-08T15:42:28.156 INFO:teuthology.orchestra.run.smithi172.stdout:Get:4 http://security.ubuntu.com/ubuntu jammy-security InRelease [110 kB] 2024-05-08T15:42:28.250 INFO:teuthology.orchestra.run.smithi172.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [1,638 kB] 2024-05-08T15:42:28.357 INFO:teuthology.orchestra.run.smithi172.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main i386 Packages [624 kB] 2024-05-08T15:42:28.364 INFO:teuthology.orchestra.run.smithi172.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main Translation-en [307 kB] 2024-05-08T15:42:28.368 INFO:teuthology.orchestra.run.smithi172.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 c-n-f Metadata [16.1 kB] 2024-05-08T15:42:28.368 INFO:teuthology.orchestra.run.smithi172.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy-updates/restricted i386 Packages [38.1 kB] 2024-05-08T15:42:28.369 INFO:teuthology.orchestra.run.smithi172.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [1,864 kB] 2024-05-08T15:42:28.393 INFO:teuthology.orchestra.run.smithi172.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy-updates/restricted Translation-en [316 kB] 2024-05-08T15:42:28.396 INFO:teuthology.orchestra.run.smithi172.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 c-n-f Metadata [520 B] 2024-05-08T15:42:28.397 INFO:teuthology.orchestra.run.smithi172.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy-updates/universe i386 Packages [702 kB] 2024-05-08T15:42:28.404 INFO:teuthology.orchestra.run.smithi172.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [1,073 kB] 2024-05-08T15:42:28.416 INFO:teuthology.orchestra.run.smithi172.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy-updates/universe Translation-en [245 kB] 2024-05-08T15:42:28.420 INFO:teuthology.orchestra.run.smithi172.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 c-n-f Metadata [22.1 kB] 2024-05-08T15:42:28.422 INFO:teuthology.orchestra.run.smithi172.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse i386 Packages [4,472 B] 2024-05-08T15:42:28.422 INFO:teuthology.orchestra.run.smithi172.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 Packages [42.7 kB] 2024-05-08T15:42:28.461 INFO:teuthology.orchestra.run.smithi172.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse Translation-en [10.4 kB] 2024-05-08T15:42:28.461 INFO:teuthology.orchestra.run.smithi172.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 c-n-f Metadata [472 B] 2024-05-08T15:42:28.462 INFO:teuthology.orchestra.run.smithi172.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-backports/main i386 Packages [59.3 kB] 2024-05-08T15:42:28.462 INFO:teuthology.orchestra.run.smithi172.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 Packages [67.1 kB] 2024-05-08T15:42:28.463 INFO:teuthology.orchestra.run.smithi172.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy-backports/main Translation-en [11.0 kB] 2024-05-08T15:42:28.463 INFO:teuthology.orchestra.run.smithi172.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 c-n-f Metadata [388 B] 2024-05-08T15:42:28.463 INFO:teuthology.orchestra.run.smithi172.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy-backports/universe i386 Packages [16.0 kB] 2024-05-08T15:42:28.463 INFO:teuthology.orchestra.run.smithi172.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 Packages [27.2 kB] 2024-05-08T15:42:28.481 INFO:teuthology.orchestra.run.smithi172.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy-backports/universe Translation-en [16.2 kB] 2024-05-08T15:42:28.501 INFO:teuthology.orchestra.run.smithi172.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 c-n-f Metadata [644 B] 2024-05-08T15:42:28.609 INFO:teuthology.orchestra.run.smithi172.stdout:Get:29 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages [1,421 kB] 2024-05-08T15:42:29.021 INFO:teuthology.orchestra.run.smithi172.stdout:Get:30 http://security.ubuntu.com/ubuntu jammy-security/main i386 Packages [458 kB] 2024-05-08T15:42:29.038 INFO:teuthology.orchestra.run.smithi172.stdout:Get:31 http://security.ubuntu.com/ubuntu jammy-security/main Translation-en [246 kB] 2024-05-08T15:42:29.048 INFO:teuthology.orchestra.run.smithi172.stdout:Get:32 http://security.ubuntu.com/ubuntu jammy-security/main amd64 c-n-f Metadata [11.4 kB] 2024-05-08T15:42:29.049 INFO:teuthology.orchestra.run.smithi172.stdout:Get:33 http://security.ubuntu.com/ubuntu jammy-security/restricted i386 Packages [36.5 kB] 2024-05-08T15:42:29.050 INFO:teuthology.orchestra.run.smithi172.stdout:Get:34 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [1,807 kB] 2024-05-08T15:42:29.143 INFO:teuthology.orchestra.run.smithi172.stdout:Get:35 http://security.ubuntu.com/ubuntu jammy-security/restricted Translation-en [307 kB] 2024-05-08T15:42:29.153 INFO:teuthology.orchestra.run.smithi172.stdout:Get:36 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 c-n-f Metadata [520 B] 2024-05-08T15:42:29.154 INFO:teuthology.orchestra.run.smithi172.stdout:Get:37 http://security.ubuntu.com/ubuntu jammy-security/universe i386 Packages [602 kB] 2024-05-08T15:42:29.173 INFO:teuthology.orchestra.run.smithi172.stdout:Get:38 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [849 kB] 2024-05-08T15:42:29.357 INFO:teuthology.orchestra.run.smithi172.stdout:Get:39 http://security.ubuntu.com/ubuntu jammy-security/universe Translation-en [163 kB] 2024-05-08T15:42:29.754 INFO:teuthology.orchestra.run.smithi172.stdout:Get:40 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 c-n-f Metadata [16.8 kB] 2024-05-08T15:42:29.761 INFO:teuthology.orchestra.run.smithi172.stdout:Get:41 http://security.ubuntu.com/ubuntu jammy-security/multiverse i386 Packages [1,356 B] 2024-05-08T15:42:29.761 INFO:teuthology.orchestra.run.smithi172.stdout:Get:42 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 Packages [37.2 kB] 2024-05-08T15:42:29.775 INFO:teuthology.orchestra.run.smithi172.stdout:Get:43 http://security.ubuntu.com/ubuntu jammy-security/multiverse Translation-en [7,588 B] 2024-05-08T15:42:29.777 INFO:teuthology.orchestra.run.smithi172.stdout:Get:44 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 c-n-f Metadata [260 B] 2024-05-08T15:42:34.028 INFO:teuthology.orchestra.run.smithi172.stdout:Fetched 13.4 MB in 2s (7,139 kB/s) 2024-05-08T15:42:35.410 INFO:teuthology.orchestra.run.smithi172.stdout:Reading package lists... 2024-05-08T15:42:35.427 DEBUG:teuthology.orchestra.run.smithi172:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-05-08T15:42:35.479 INFO:teuthology.orchestra.run.smithi172.stdout:Reading package lists... 2024-05-08T15:42:35.657 INFO:teuthology.orchestra.run.smithi172.stdout:Building dependency tree... 2024-05-08T15:42:35.658 INFO:teuthology.orchestra.run.smithi172.stdout:Reading state information... 2024-05-08T15:42:35.814 INFO:teuthology.orchestra.run.smithi172.stdout:The following packages were automatically installed and are no longer required: 2024-05-08T15:42:35.814 INFO:teuthology.orchestra.run.smithi172.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-05-08T15:42:35.814 INFO:teuthology.orchestra.run.smithi172.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-05-08T15:42:35.814 INFO:teuthology.orchestra.run.smithi172.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-05-08T15:42:35.814 INFO:teuthology.orchestra.run.smithi172.stdout: python2 python2-dev python2-minimal python2.7 python2.7-dev 2024-05-08T15:42:35.815 INFO:teuthology.orchestra.run.smithi172.stdout: python2.7-minimal 2024-05-08T15:42:35.815 INFO:teuthology.orchestra.run.smithi172.stdout:Use 'sudo apt autoremove' to remove them. 2024-05-08T15:42:35.815 INFO:teuthology.orchestra.run.smithi172.stdout:The following additional packages will be installed: 2024-05-08T15:42:35.815 INFO:teuthology.orchestra.run.smithi172.stdout: linux-generic linux-headers-5.15.0-106 linux-headers-5.15.0-106-generic 2024-05-08T15:42:35.815 INFO:teuthology.orchestra.run.smithi172.stdout: linux-headers-generic linux-image-5.15.0-106-generic 2024-05-08T15:42:35.816 INFO:teuthology.orchestra.run.smithi172.stdout: linux-modules-5.15.0-106-generic linux-modules-extra-5.15.0-106-generic 2024-05-08T15:42:35.817 INFO:teuthology.orchestra.run.smithi172.stdout:Suggested packages: 2024-05-08T15:42:35.817 INFO:teuthology.orchestra.run.smithi172.stdout: fdutils linux-doc | linux-source-5.15.0 linux-tools 2024-05-08T15:42:35.817 INFO:teuthology.orchestra.run.smithi172.stdout:Recommended packages: 2024-05-08T15:42:35.817 INFO:teuthology.orchestra.run.smithi172.stdout: thermald 2024-05-08T15:42:35.863 INFO:teuthology.orchestra.run.smithi172.stdout:The following NEW packages will be installed: 2024-05-08T15:42:35.863 INFO:teuthology.orchestra.run.smithi172.stdout: linux-headers-5.15.0-106 linux-headers-5.15.0-106-generic 2024-05-08T15:42:35.863 INFO:teuthology.orchestra.run.smithi172.stdout: linux-image-5.15.0-106-generic linux-modules-5.15.0-106-generic 2024-05-08T15:42:35.864 INFO:teuthology.orchestra.run.smithi172.stdout: linux-modules-extra-5.15.0-106-generic 2024-05-08T15:42:35.865 INFO:teuthology.orchestra.run.smithi172.stdout:The following packages will be upgraded: 2024-05-08T15:42:35.865 INFO:teuthology.orchestra.run.smithi172.stdout: linux-generic linux-headers-generic linux-image-generic 2024-05-08T15:42:35.923 INFO:teuthology.orchestra.run.smithi172.stdout:3 upgraded, 5 newly installed, 0 to remove and 316 not upgraded. 2024-05-08T15:42:35.923 INFO:teuthology.orchestra.run.smithi172.stdout:Need to get 113 MB of archives. 2024-05-08T15:42:35.923 INFO:teuthology.orchestra.run.smithi172.stdout:After this operation, 583 MB of additional disk space will be used. 2024-05-08T15:42:35.923 INFO:teuthology.orchestra.run.smithi172.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-5.15.0-106-generic amd64 5.15.0-106.116 [22.7 MB] 2024-05-08T15:42:36.490 INFO:teuthology.orchestra.run.smithi172.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-5.15.0-106-generic amd64 5.15.0-106.116 [11.5 MB] 2024-05-08T15:42:36.634 INFO:teuthology.orchestra.run.smithi172.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-extra-5.15.0-106-generic amd64 5.15.0-106.116 [63.9 MB] 2024-05-08T15:42:37.935 INFO:teuthology.orchestra.run.smithi172.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-generic amd64 5.15.0.106.106 [1,692 B] 2024-05-08T15:42:37.935 INFO:teuthology.orchestra.run.smithi172.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-generic amd64 5.15.0.106.106 [2,502 B] 2024-05-08T15:42:37.935 INFO:teuthology.orchestra.run.smithi172.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-106 all 5.15.0-106.116 [12.3 MB] 2024-05-08T15:42:38.126 INFO:teuthology.orchestra.run.smithi172.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-106-generic amd64 5.15.0-106.116 [2,876 kB] 2024-05-08T15:42:38.157 INFO:teuthology.orchestra.run.smithi172.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-generic amd64 5.15.0.106.106 [2,366 B] 2024-05-08T15:42:38.541 INFO:teuthology.orchestra.run.smithi172.stdout:Fetched 113 MB in 2s (49.6 MB/s) 2024-05-08T15:42:38.719 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package linux-modules-5.15.0-106-generic. 2024-05-08T15:42:40.754 INFO:teuthology.orchestra.run.smithi172.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 129542 files and directories currently installed.) 2024-05-08T15:42:40.759 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../0-linux-modules-5.15.0-106-generic_5.15.0-106.116_amd64.deb ... 2024-05-08T15:42:40.965 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking linux-modules-5.15.0-106-generic (5.15.0-106.116) ... 2024-05-08T15:42:43.007 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package linux-image-5.15.0-106-generic. 2024-05-08T15:42:43.029 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../1-linux-image-5.15.0-106-generic_5.15.0-106.116_amd64.deb ... 2024-05-08T15:42:43.127 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking linux-image-5.15.0-106-generic (5.15.0-106.116) ... 2024-05-08T15:42:43.536 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package linux-modules-extra-5.15.0-106-generic. 2024-05-08T15:42:43.556 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../2-linux-modules-extra-5.15.0-106-generic_5.15.0-106.116_amd64.deb ... 2024-05-08T15:42:43.585 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking linux-modules-extra-5.15.0-106-generic (5.15.0-106.116) ... 2024-05-08T15:42:50.460 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../3-linux-generic_5.15.0.106.106_amd64.deb ... 2024-05-08T15:42:50.585 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking linux-generic (5.15.0.106.106) over (5.15.0.56.54) ... 2024-05-08T15:42:50.950 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../4-linux-image-generic_5.15.0.106.106_amd64.deb ... 2024-05-08T15:42:51.065 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking linux-image-generic (5.15.0.106.106) over (5.15.0.56.54) ... 2024-05-08T15:42:51.351 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package linux-headers-5.15.0-106. 2024-05-08T15:42:51.373 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../5-linux-headers-5.15.0-106_5.15.0-106.116_all.deb ... 2024-05-08T15:42:51.418 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking linux-headers-5.15.0-106 (5.15.0-106.116) ... 2024-05-08T15:42:57.301 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package linux-headers-5.15.0-106-generic. 2024-05-08T15:42:57.338 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../6-linux-headers-5.15.0-106-generic_5.15.0-106.116_amd64.deb ... 2024-05-08T15:42:57.377 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking linux-headers-5.15.0-106-generic (5.15.0-106.116) ... 2024-05-08T15:42:59.937 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../7-linux-headers-generic_5.15.0.106.106_amd64.deb ... 2024-05-08T15:43:00.044 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking linux-headers-generic (5.15.0.106.106) over (5.15.0.56.54) ... 2024-05-08T15:43:00.361 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up linux-headers-5.15.0-106 (5.15.0-106.116) ... 2024-05-08T15:43:00.490 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up linux-headers-5.15.0-106-generic (5.15.0-106.116) ... 2024-05-08T15:43:00.624 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up linux-headers-generic (5.15.0.106.106) ... 2024-05-08T15:43:00.753 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up linux-image-5.15.0-106-generic (5.15.0-106.116) ... 2024-05-08T15:43:02.077 INFO:teuthology.orchestra.run.smithi172.stdout:I: /vmlinuz.old is now a symlink to boot/vmlinuz-5.15.0-56-generic 2024-05-08T15:43:02.077 INFO:teuthology.orchestra.run.smithi172.stdout:I: /initrd.img.old is now a symlink to boot/initrd.img-5.15.0-56-generic 2024-05-08T15:43:02.077 INFO:teuthology.orchestra.run.smithi172.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.15.0-106-generic 2024-05-08T15:43:02.077 INFO:teuthology.orchestra.run.smithi172.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.15.0-106-generic 2024-05-08T15:43:02.238 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up linux-modules-5.15.0-106-generic (5.15.0-106.116) ... 2024-05-08T15:43:04.348 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up linux-modules-extra-5.15.0-106-generic (5.15.0-106.116) ... 2024-05-08T15:43:05.774 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up linux-image-generic (5.15.0.106.106) ... 2024-05-08T15:43:05.900 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up linux-generic (5.15.0.106.106) ... 2024-05-08T15:43:06.026 INFO:teuthology.orchestra.run.smithi172.stdout:Processing triggers for linux-image-5.15.0-106-generic (5.15.0-106.116) ... 2024-05-08T15:43:06.089 INFO:teuthology.orchestra.run.smithi172.stdout:/etc/kernel/postinst.d/initramfs-tools: 2024-05-08T15:43:06.089 INFO:teuthology.orchestra.run.smithi172.stdout:update-initramfs: Generating /boot/initrd.img-5.15.0-106-generic 2024-05-08T15:43:20.290 INFO:teuthology.orchestra.run.smithi172.stdout:/etc/kernel/postinst.d/zz-update-grub: 2024-05-08T15:43:20.291 INFO:teuthology.orchestra.run.smithi172.stdout:Sourcing file `/etc/default/grub' 2024-05-08T15:43:20.301 INFO:teuthology.orchestra.run.smithi172.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-05-08T15:43:20.325 INFO:teuthology.orchestra.run.smithi172.stdout:Generating grub configuration file ... 2024-05-08T15:43:20.618 INFO:teuthology.orchestra.run.smithi172.stdout:Found linux image: /boot/vmlinuz-5.15.0-106-generic 2024-05-08T15:43:20.645 INFO:teuthology.orchestra.run.smithi172.stdout:Found initrd image: /boot/initrd.img-5.15.0-106-generic 2024-05-08T15:43:20.914 INFO:teuthology.orchestra.run.smithi172.stdout:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-05-08T15:43:20.919 INFO:teuthology.orchestra.run.smithi172.stdout:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-05-08T15:43:20.993 INFO:teuthology.orchestra.run.smithi172.stdout:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-05-08T15:43:20.998 INFO:teuthology.orchestra.run.smithi172.stdout:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-05-08T15:43:21.168 INFO:teuthology.orchestra.run.smithi172.stdout:Warning: os-prober will not be executed to detect other bootable partitions. 2024-05-08T15:43:21.168 INFO:teuthology.orchestra.run.smithi172.stdout:Systems on them will not be added to the GRUB boot configuration. 2024-05-08T15:43:21.168 INFO:teuthology.orchestra.run.smithi172.stdout:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-05-08T15:43:21.206 INFO:teuthology.orchestra.run.smithi172.stdout:done 2024-05-08T15:43:21.663 DEBUG:teuthology.orchestra.run.smithi172:> dpkg -s linux-image-generic 2024-05-08T15:43:21.679 INFO:teuthology.orchestra.run.smithi172.stdout:Package: linux-image-generic 2024-05-08T15:43:21.680 INFO:teuthology.orchestra.run.smithi172.stdout:Status: install ok installed 2024-05-08T15:43:21.680 INFO:teuthology.orchestra.run.smithi172.stdout:Priority: optional 2024-05-08T15:43:21.680 INFO:teuthology.orchestra.run.smithi172.stdout:Section: kernel 2024-05-08T15:43:21.680 INFO:teuthology.orchestra.run.smithi172.stdout:Installed-Size: 21 2024-05-08T15:43:21.680 INFO:teuthology.orchestra.run.smithi172.stdout:Maintainer: Ubuntu Kernel Team 2024-05-08T15:43:21.680 INFO:teuthology.orchestra.run.smithi172.stdout:Architecture: amd64 2024-05-08T15:43:21.680 INFO:teuthology.orchestra.run.smithi172.stdout:Source: linux-meta 2024-05-08T15:43:21.680 INFO:teuthology.orchestra.run.smithi172.stdout:Version: 5.15.0.106.106 2024-05-08T15:43:21.680 INFO:teuthology.orchestra.run.smithi172.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.3), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-106), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.3) 2024-05-08T15:43:21.680 INFO:teuthology.orchestra.run.smithi172.stdout:Depends: linux-image-5.15.0-106-generic, linux-modules-extra-5.15.0-106-generic, linux-firmware, intel-microcode, amd64-microcode 2024-05-08T15:43:21.680 INFO:teuthology.orchestra.run.smithi172.stdout:Recommends: thermald 2024-05-08T15:43:21.680 INFO:teuthology.orchestra.run.smithi172.stdout:Description: Generic Linux kernel image 2024-05-08T15:43:21.680 INFO:teuthology.orchestra.run.smithi172.stdout: This package will always depend on the latest generic kernel image 2024-05-08T15:43:21.680 INFO:teuthology.orchestra.run.smithi172.stdout: available. 2024-05-08T15:43:21.681 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.15.0-56-generic Expected: 5.15.0-106-generic 2024-05-08T15:43:21.681 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-05-08T15:43:21.681 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-05-08T15:43:21.681 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-05-08T15:43:21.681 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi172.front.sepia.ceph.com, path=None, version=distro) 2024-05-08T15:43:21.681 DEBUG:teuthology.orchestra.run.smithi172:> sudo apt-get clean 2024-05-08T15:43:21.848 DEBUG:teuthology.orchestra.run.smithi172:> sudo apt-get update 2024-05-08T15:43:21.984 INFO:teuthology.orchestra.run.smithi172.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-05-08T15:43:22.001 INFO:teuthology.orchestra.run.smithi172.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-05-08T15:43:22.021 INFO:teuthology.orchestra.run.smithi172.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-05-08T15:43:22.096 INFO:teuthology.orchestra.run.smithi172.stdout:Hit:4 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-05-08T15:43:23.418 INFO:teuthology.orchestra.run.smithi172.stdout:Reading package lists... 2024-05-08T15:43:23.436 DEBUG:teuthology.orchestra.run.smithi172:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-05-08T15:43:23.497 INFO:teuthology.orchestra.run.smithi172.stdout:Reading package lists... 2024-05-08T15:43:23.693 INFO:teuthology.orchestra.run.smithi172.stdout:Building dependency tree... 2024-05-08T15:43:23.693 INFO:teuthology.orchestra.run.smithi172.stdout:Reading state information... 2024-05-08T15:43:23.872 INFO:teuthology.orchestra.run.smithi172.stdout:linux-image-generic is already the newest version (5.15.0.106.106). 2024-05-08T15:43:23.872 INFO:teuthology.orchestra.run.smithi172.stdout:The following packages were automatically installed and are no longer required: 2024-05-08T15:43:23.872 INFO:teuthology.orchestra.run.smithi172.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-05-08T15:43:23.872 INFO:teuthology.orchestra.run.smithi172.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-05-08T15:43:23.873 INFO:teuthology.orchestra.run.smithi172.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-05-08T15:43:23.873 INFO:teuthology.orchestra.run.smithi172.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-05-08T15:43:23.873 INFO:teuthology.orchestra.run.smithi172.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-05-08T15:43:23.873 INFO:teuthology.orchestra.run.smithi172.stdout:Use 'sudo apt autoremove' to remove them. 2024-05-08T15:43:23.913 INFO:teuthology.orchestra.run.smithi172.stdout:0 upgraded, 0 newly installed, 0 to remove and 316 not upgraded. 2024-05-08T15:43:23.915 DEBUG:teuthology.orchestra.run.smithi172:> dpkg -s linux-image-generic 2024-05-08T15:43:23.927 INFO:teuthology.orchestra.run.smithi172.stdout:Package: linux-image-generic 2024-05-08T15:43:23.927 INFO:teuthology.orchestra.run.smithi172.stdout:Status: install ok installed 2024-05-08T15:43:23.927 INFO:teuthology.orchestra.run.smithi172.stdout:Priority: optional 2024-05-08T15:43:23.927 INFO:teuthology.orchestra.run.smithi172.stdout:Section: kernel 2024-05-08T15:43:23.927 INFO:teuthology.orchestra.run.smithi172.stdout:Installed-Size: 21 2024-05-08T15:43:23.928 INFO:teuthology.orchestra.run.smithi172.stdout:Maintainer: Ubuntu Kernel Team 2024-05-08T15:43:23.928 INFO:teuthology.orchestra.run.smithi172.stdout:Architecture: amd64 2024-05-08T15:43:23.928 INFO:teuthology.orchestra.run.smithi172.stdout:Source: linux-meta 2024-05-08T15:43:23.928 INFO:teuthology.orchestra.run.smithi172.stdout:Version: 5.15.0.106.106 2024-05-08T15:43:23.928 INFO:teuthology.orchestra.run.smithi172.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.3), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-106), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.3) 2024-05-08T15:43:23.928 INFO:teuthology.orchestra.run.smithi172.stdout:Depends: linux-image-5.15.0-106-generic, linux-modules-extra-5.15.0-106-generic, linux-firmware, intel-microcode, amd64-microcode 2024-05-08T15:43:23.928 INFO:teuthology.orchestra.run.smithi172.stdout:Recommends: thermald 2024-05-08T15:43:23.928 INFO:teuthology.orchestra.run.smithi172.stdout:Description: Generic Linux kernel image 2024-05-08T15:43:23.928 INFO:teuthology.orchestra.run.smithi172.stdout: This package will always depend on the latest generic kernel image 2024-05-08T15:43:23.928 INFO:teuthology.orchestra.run.smithi172.stdout: available. 2024-05-08T15:43:23.928 DEBUG:teuthology.orchestra.run.smithi172:> mktemp 2024-05-08T15:43:23.975 INFO:teuthology.orchestra.run.smithi172.stdout:/tmp/tmp.vXK3jFT2qZ 2024-05-08T15:43:23.975 DEBUG:teuthology.orchestra.run.smithi172:> sudo cp /boot/grub/grub.cfg /tmp/tmp.vXK3jFT2qZ 2024-05-08T15:43:24.030 DEBUG:teuthology.orchestra.run.smithi172:> sudo chmod 0666 /tmp/tmp.vXK3jFT2qZ 2024-05-08T15:43:24.150 DEBUG:teuthology.orchestra.remote:smithi172:/tmp/tmp.vXK3jFT2qZ is 10KB 2024-05-08T15:43:24.165 DEBUG:teuthology.orchestra.run.smithi172:> rm -fr /tmp/tmp.vXK3jFT2qZ 2024-05-08T15:43:24.171 DEBUG:teuthology.orchestra.run.smithi172:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2024-05-08T15:43:24.227 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2024-05-08T15:43:24.227 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd of=/etc/grub.d/01_ceph_kernel 2024-05-08T15:43:24.227 DEBUG:teuthology.orchestra.run.smithi172:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2024-05-08T15:43:24.295 INFO:teuthology.task.kernel:Distro Kernel Version: 5.15.0-106-generic 2024-05-08T15:43:24.296 DEBUG:teuthology.orchestra.run.smithi172:> sudo update-grub 2024-05-08T15:43:25.424 INFO:teuthology.orchestra.run.smithi172.stderr:Sourcing file `/etc/default/grub' 2024-05-08T15:43:25.425 INFO:teuthology.orchestra.run.smithi172.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-05-08T15:43:25.429 INFO:teuthology.orchestra.run.smithi172.stderr:Generating grub configuration file ... 2024-05-08T15:43:25.597 INFO:teuthology.orchestra.run.smithi172.stderr:Found linux image: /boot/vmlinuz-5.15.0-106-generic 2024-05-08T15:43:25.602 INFO:teuthology.orchestra.run.smithi172.stderr:Found initrd image: /boot/initrd.img-5.15.0-106-generic 2024-05-08T15:43:25.837 INFO:teuthology.orchestra.run.smithi172.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-05-08T15:43:25.842 INFO:teuthology.orchestra.run.smithi172.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-05-08T15:43:25.884 INFO:teuthology.orchestra.run.smithi172.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-05-08T15:43:25.889 INFO:teuthology.orchestra.run.smithi172.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-05-08T15:43:26.008 INFO:teuthology.orchestra.run.smithi172.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2024-05-08T15:43:26.008 INFO:teuthology.orchestra.run.smithi172.stderr:Systems on them will not be added to the GRUB boot configuration. 2024-05-08T15:43:26.008 INFO:teuthology.orchestra.run.smithi172.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-05-08T15:43:26.038 INFO:teuthology.orchestra.run.smithi172.stderr:done 2024-05-08T15:43:26.047 DEBUG:teuthology.orchestra.run.smithi172:> sudo shutdown -r now 2024-05-08T15:43:56.078 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-05-08T15:43:56.079 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi172.front.sepia.ceph.com' 2024-05-08T15:43:56.080 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi172.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-08T15:44:14.623 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.172 2024-05-08T15:44:23.631 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi172.front.sepia.ceph.com' 2024-05-08T15:44:23.632 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi172.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-08T15:44:26.686 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.172 2024-05-08T15:44:38.698 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi172.front.sepia.ceph.com' 2024-05-08T15:44:38.699 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi172.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-08T15:45:38.759 DEBUG:teuthology.orchestra.remote:timed out 2024-05-08T15:45:53.774 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi172.front.sepia.ceph.com' 2024-05-08T15:45:53.775 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi172.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-08T15:45:54.165 DEBUG:teuthology.orchestra.run.smithi172:> true 2024-05-08T15:45:54.785 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi172.front.sepia.ceph.com' 2024-05-08T15:45:54.785 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.15.0-106-generic"... 2024-05-08T15:45:54.785 DEBUG:teuthology.orchestra.run.smithi172:> uname -r 2024-05-08T15:45:54.839 INFO:teuthology.orchestra.run.smithi172.stdout:5.15.0-106-generic 2024-05-08T15:45:54.839 DEBUG:teuthology.task.kernel:current kernel version is 5.15.0-106-generic vs 5.15.0-106-generic 2024-05-08T15:45:54.839 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-05-08T15:45:54.839 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2024-05-08T15:45:55.840 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-05-08T15:45:55.840 DEBUG:teuthology.orchestra.run.smithi172:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-05-08T15:45:55.909 INFO:teuthology.orchestra.run.smithi172.stdout:ttyS1 2024-05-08T15:45:55.931 DEBUG:teuthology.parallel:result is None 2024-05-08T15:45:55.931 INFO:teuthology.run_tasks:Running task internal.base... 2024-05-08T15:45:55.939 INFO:teuthology.task.internal:Creating test directory... 2024-05-08T15:45:55.940 DEBUG:teuthology.orchestra.run.smithi172:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-05-08T15:45:55.945 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-05-08T15:45:55.981 INFO:teuthology.run_tasks:Running task internal.archive... 2024-05-08T15:45:56.015 INFO:teuthology.task.internal:Creating archive directory... 2024-05-08T15:45:56.015 DEBUG:teuthology.orchestra.run.smithi172:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-05-08T15:45:56.029 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-05-08T15:45:56.048 INFO:teuthology.task.internal:Enabling coredump saving... 2024-05-08T15:45:56.049 DEBUG:teuthology.orchestra.run.smithi172:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-05-08T15:45:56.089 INFO:teuthology.orchestra.run.smithi172.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-08T15:45:56.097 INFO:teuthology.orchestra.run.smithi172.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-08T15:45:56.098 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-05-08T15:45:56.106 INFO:teuthology.task.internal:Configuring sudo... 2024-05-08T15:45:56.106 DEBUG:teuthology.orchestra.run.smithi172:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-05-08T15:45:56.158 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-05-08T15:45:56.167 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-05-08T15:45:56.167 DEBUG:teuthology.orchestra.run.smithi172:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-05-08T15:45:56.206 DEBUG:teuthology.orchestra.run.smithi172:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-05-08T15:45:56.254 DEBUG:teuthology.orchestra.run.smithi172:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-05-08T15:45:56.302 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2024-05-08T15:45:56.302 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-05-08T15:45:56.373 DEBUG:teuthology.orchestra.run.smithi172:> sudo service rsyslog restart 2024-05-08T15:45:56.449 INFO:teuthology.run_tasks:Running task internal.timer... 2024-05-08T15:45:56.457 INFO:teuthology.task.internal:Starting timer... 2024-05-08T15:45:56.457 INFO:teuthology.run_tasks:Running task pcp... 2024-05-08T15:45:56.485 INFO:teuthology.run_tasks:Running task selinux... 2024-05-08T15:45:56.519 DEBUG:teuthology.task.selinux:Excluding smithi172: OS 'ubuntu' does not support SELinux 2024-05-08T15:45:56.519 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-05-08T15:45:56.519 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2024-05-08T15:45:56.520 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-05-08T15:45:56.520 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-05-08T15:45:56.552 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-05-08T15:45:56.557 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-05-08T15:45:56.682 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-05-08T15:45:56.697 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-05-08T15:45:56.699 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi172.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-05-08T15:52:51.017 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi172.front.sepia.ceph.com')] 2024-05-08T15:52:51.018 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi172.front.sepia.ceph.com' 2024-05-08T15:52:51.019 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi172.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-08T15:52:51.099 DEBUG:teuthology.orchestra.run.smithi172:> true 2024-05-08T15:52:51.178 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi172.front.sepia.ceph.com' 2024-05-08T15:52:51.178 INFO:teuthology.run_tasks:Running task clock... 2024-05-08T15:52:51.190 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-05-08T15:52:51.190 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-05-08T15:52:51.190 DEBUG:teuthology.orchestra.run.smithi172:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-05-08T15:52:51.244 INFO:teuthology.orchestra.run.smithi172.stdout: 8 May 15:52:51 ntpd[17263]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2024-05-08T15:52:51.245 INFO:teuthology.orchestra.run.smithi172.stdout: 8 May 15:52:51 ntpd[17263]: Command line: ntpd -gq 2024-05-08T15:52:51.245 INFO:teuthology.orchestra.run.smithi172.stdout: 8 May 15:52:51 ntpd[17263]: ---------------------------------------------------- 2024-05-08T15:52:51.245 INFO:teuthology.orchestra.run.smithi172.stdout: 8 May 15:52:51 ntpd[17263]: ntp-4 is maintained by Network Time Foundation, 2024-05-08T15:52:51.245 INFO:teuthology.orchestra.run.smithi172.stdout: 8 May 15:52:51 ntpd[17263]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2024-05-08T15:52:51.245 INFO:teuthology.orchestra.run.smithi172.stdout: 8 May 15:52:51 ntpd[17263]: corporation. Support and training for ntp-4 are 2024-05-08T15:52:51.246 INFO:teuthology.orchestra.run.smithi172.stdout: 8 May 15:52:51 ntpd[17263]: available at https://www.nwtime.org/support 2024-05-08T15:52:51.246 INFO:teuthology.orchestra.run.smithi172.stdout: 8 May 15:52:51 ntpd[17263]: ---------------------------------------------------- 2024-05-08T15:52:51.247 INFO:teuthology.orchestra.run.smithi172.stdout: 8 May 15:52:51 ntpd[17263]: proto: precision = 0.060 usec (-24) 2024-05-08T15:52:51.247 INFO:teuthology.orchestra.run.smithi172.stdout: 8 May 15:52:51 ntpd[17263]: basedate set to 2022-02-04 2024-05-08T15:52:51.247 INFO:teuthology.orchestra.run.smithi172.stdout: 8 May 15:52:51 ntpd[17263]: gps base set to 2022-02-06 (week 2196) 2024-05-08T15:52:51.248 INFO:teuthology.orchestra.run.smithi172.stdout: 8 May 15:52:51 ntpd[17263]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-05-08T15:52:51.248 INFO:teuthology.orchestra.run.smithi172.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-05-08T15:52:51.248 INFO:teuthology.orchestra.run.smithi172.stderr:restrict ::: KOD does nothing without LIMITED. 2024-05-08T15:52:51.249 INFO:teuthology.orchestra.run.smithi172.stdout: 8 May 15:52:51 ntpd[17263]: restrict ::: KOD does nothing without LIMITED. 2024-05-08T15:52:51.249 INFO:teuthology.orchestra.run.smithi172.stdout: 8 May 15:52:51 ntpd[17263]: Listen and drop on 0 v6wildcard [::]:123 2024-05-08T15:52:51.249 INFO:teuthology.orchestra.run.smithi172.stdout: 8 May 15:52:51 ntpd[17263]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2024-05-08T15:52:51.249 INFO:teuthology.orchestra.run.smithi172.stdout: 8 May 15:52:51 ntpd[17263]: Listen normally on 2 lo 127.0.0.1:123 2024-05-08T15:52:51.250 INFO:teuthology.orchestra.run.smithi172.stdout: 8 May 15:52:51 ntpd[17263]: Listen normally on 3 enp3s0f1 172.21.15.172:123 2024-05-08T15:52:51.250 INFO:teuthology.orchestra.run.smithi172.stdout: 8 May 15:52:51 ntpd[17263]: Listen normally on 4 lo [::1]:123 2024-05-08T15:52:51.250 INFO:teuthology.orchestra.run.smithi172.stdout: 8 May 15:52:51 ntpd[17263]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe88:7f91%5]:123 2024-05-08T15:52:51.250 INFO:teuthology.orchestra.run.smithi172.stdout: 8 May 15:52:51 ntpd[17263]: Listening on routing socket on fd #22 for interface updates 2024-05-08T15:52:52.245 INFO:teuthology.orchestra.run.smithi172.stderr: 8 May 15:52:52 ntpd[17263]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-05-08T15:52:52.245 INFO:teuthology.orchestra.run.smithi172.stderr: 8 May 15:52:52 ntpd[17263]: can't open /var/log/ntpstats/rawstats.20240508: Permission denied 2024-05-08T15:52:52.245 INFO:teuthology.orchestra.run.smithi172.stderr: 8 May 15:52:52 ntpd[17263]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-05-08T15:52:52.245 INFO:teuthology.orchestra.run.smithi172.stderr: 8 May 15:52:52 ntpd[17263]: can't open /var/log/ntpstats/rawstats.20240508: Permission denied 2024-05-08T15:52:52.245 INFO:teuthology.orchestra.run.smithi172.stderr: 8 May 15:52:52 ntpd[17263]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-05-08T15:52:52.245 INFO:teuthology.orchestra.run.smithi172.stderr: 8 May 15:52:52 ntpd[17263]: can't open /var/log/ntpstats/rawstats.20240508: Permission denied 2024-05-08T15:52:52.246 INFO:teuthology.orchestra.run.smithi172.stderr: 8 May 15:52:52 ntpd[17263]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-05-08T15:52:52.246 INFO:teuthology.orchestra.run.smithi172.stderr: 8 May 15:52:52 ntpd[17263]: can't open /var/log/ntpstats/peerstats.20240508: Permission denied 2024-05-08T15:52:53.245 INFO:teuthology.orchestra.run.smithi172.stderr: 8 May 15:52:53 ntpd[17263]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-05-08T15:52:53.246 INFO:teuthology.orchestra.run.smithi172.stderr: 8 May 15:52:53 ntpd[17263]: can't open /var/log/ntpstats/rawstats.20240508: Permission denied 2024-05-08T15:52:53.246 INFO:teuthology.orchestra.run.smithi172.stderr: 8 May 15:52:53 ntpd[17263]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-05-08T15:52:53.246 INFO:teuthology.orchestra.run.smithi172.stderr: 8 May 15:52:53 ntpd[17263]: can't open /var/log/ntpstats/rawstats.20240508: Permission denied 2024-05-08T15:52:53.246 INFO:teuthology.orchestra.run.smithi172.stderr: 8 May 15:52:53 ntpd[17263]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-05-08T15:52:53.246 INFO:teuthology.orchestra.run.smithi172.stderr: 8 May 15:52:53 ntpd[17263]: can't open /var/log/ntpstats/peerstats.20240508: Permission denied 2024-05-08T15:52:54.245 INFO:teuthology.orchestra.run.smithi172.stderr: 8 May 15:52:54 ntpd[17263]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-05-08T15:52:54.245 INFO:teuthology.orchestra.run.smithi172.stderr: 8 May 15:52:54 ntpd[17263]: can't open /var/log/ntpstats/rawstats.20240508: Permission denied 2024-05-08T15:52:54.245 INFO:teuthology.orchestra.run.smithi172.stderr: 8 May 15:52:54 ntpd[17263]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-05-08T15:52:54.246 INFO:teuthology.orchestra.run.smithi172.stderr: 8 May 15:52:54 ntpd[17263]: can't open /var/log/ntpstats/rawstats.20240508: Permission denied 2024-05-08T15:52:54.246 INFO:teuthology.orchestra.run.smithi172.stderr: 8 May 15:52:54 ntpd[17263]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-05-08T15:52:54.246 INFO:teuthology.orchestra.run.smithi172.stderr: 8 May 15:52:54 ntpd[17263]: can't open /var/log/ntpstats/peerstats.20240508: Permission denied 2024-05-08T15:52:55.245 INFO:teuthology.orchestra.run.smithi172.stderr: 8 May 15:52:55 ntpd[17263]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-05-08T15:52:55.245 INFO:teuthology.orchestra.run.smithi172.stderr: 8 May 15:52:55 ntpd[17263]: can't open /var/log/ntpstats/rawstats.20240508: Permission denied 2024-05-08T15:52:55.245 INFO:teuthology.orchestra.run.smithi172.stderr: 8 May 15:52:55 ntpd[17263]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-05-08T15:52:55.245 INFO:teuthology.orchestra.run.smithi172.stderr: 8 May 15:52:55 ntpd[17263]: can't open /var/log/ntpstats/rawstats.20240508: Permission denied 2024-05-08T15:52:55.245 INFO:teuthology.orchestra.run.smithi172.stderr: 8 May 15:52:55 ntpd[17263]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-05-08T15:52:55.245 INFO:teuthology.orchestra.run.smithi172.stderr: 8 May 15:52:55 ntpd[17263]: can't open /var/log/ntpstats/peerstats.20240508: Permission denied 2024-05-08T15:52:56.246 INFO:teuthology.orchestra.run.smithi172.stderr: 8 May 15:52:56 ntpd[17263]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-05-08T15:52:56.246 INFO:teuthology.orchestra.run.smithi172.stderr: 8 May 15:52:56 ntpd[17263]: can't open /var/log/ntpstats/rawstats.20240508: Permission denied 2024-05-08T15:52:56.246 INFO:teuthology.orchestra.run.smithi172.stderr: 8 May 15:52:56 ntpd[17263]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-05-08T15:52:56.246 INFO:teuthology.orchestra.run.smithi172.stderr: 8 May 15:52:56 ntpd[17263]: can't open /var/log/ntpstats/rawstats.20240508: Permission denied 2024-05-08T15:52:56.246 INFO:teuthology.orchestra.run.smithi172.stderr: 8 May 15:52:56 ntpd[17263]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-05-08T15:52:56.246 INFO:teuthology.orchestra.run.smithi172.stderr: 8 May 15:52:56 ntpd[17263]: can't open /var/log/ntpstats/rawstats.20240508: Permission denied 2024-05-08T15:52:56.246 INFO:teuthology.orchestra.run.smithi172.stderr: 8 May 15:52:56 ntpd[17263]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-05-08T15:52:56.246 INFO:teuthology.orchestra.run.smithi172.stderr: 8 May 15:52:56 ntpd[17263]: can't open /var/log/ntpstats/peerstats.20240508: Permission denied 2024-05-08T15:52:56.246 INFO:teuthology.orchestra.run.smithi172.stderr: 8 May 15:52:56 ntpd[17263]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-05-08T15:52:56.246 INFO:teuthology.orchestra.run.smithi172.stderr: 8 May 15:52:56 ntpd[17263]: can't open /var/log/ntpstats/rawstats.20240508: Permission denied 2024-05-08T15:52:56.246 INFO:teuthology.orchestra.run.smithi172.stderr: 8 May 15:52:56 ntpd[17263]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-05-08T15:52:56.247 INFO:teuthology.orchestra.run.smithi172.stderr: 8 May 15:52:56 ntpd[17263]: can't open /var/log/ntpstats/peerstats.20240508: Permission denied 2024-05-08T15:52:57.245 INFO:teuthology.orchestra.run.smithi172.stderr: 8 May 15:52:57 ntpd[17263]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-05-08T15:52:57.245 INFO:teuthology.orchestra.run.smithi172.stderr: 8 May 15:52:57 ntpd[17263]: can't open /var/log/ntpstats/rawstats.20240508: Permission denied 2024-05-08T15:52:57.245 INFO:teuthology.orchestra.run.smithi172.stderr: 8 May 15:52:57 ntpd[17263]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-05-08T15:52:57.245 INFO:teuthology.orchestra.run.smithi172.stderr: 8 May 15:52:57 ntpd[17263]: can't open /var/log/ntpstats/rawstats.20240508: Permission denied 2024-05-08T15:52:57.245 INFO:teuthology.orchestra.run.smithi172.stderr: 8 May 15:52:57 ntpd[17263]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-05-08T15:52:57.245 INFO:teuthology.orchestra.run.smithi172.stderr: 8 May 15:52:57 ntpd[17263]: can't open /var/log/ntpstats/peerstats.20240508: Permission denied 2024-05-08T15:52:58.245 INFO:teuthology.orchestra.run.smithi172.stderr: 8 May 15:52:58 ntpd[17263]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-05-08T15:52:58.245 INFO:teuthology.orchestra.run.smithi172.stderr: 8 May 15:52:58 ntpd[17263]: can't open /var/log/ntpstats/rawstats.20240508: Permission denied 2024-05-08T15:52:58.246 INFO:teuthology.orchestra.run.smithi172.stderr: 8 May 15:52:58 ntpd[17263]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-05-08T15:52:58.246 INFO:teuthology.orchestra.run.smithi172.stderr: 8 May 15:52:58 ntpd[17263]: can't open /var/log/ntpstats/rawstats.20240508: Permission denied 2024-05-08T15:52:58.246 INFO:teuthology.orchestra.run.smithi172.stderr: 8 May 15:52:58 ntpd[17263]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-05-08T15:52:58.246 INFO:teuthology.orchestra.run.smithi172.stderr: 8 May 15:52:58 ntpd[17263]: can't open /var/log/ntpstats/rawstats.20240508: Permission denied 2024-05-08T15:52:58.246 INFO:teuthology.orchestra.run.smithi172.stderr: 8 May 15:52:58 ntpd[17263]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-05-08T15:52:58.246 INFO:teuthology.orchestra.run.smithi172.stderr: 8 May 15:52:58 ntpd[17263]: can't open /var/log/ntpstats/peerstats.20240508: Permission denied 2024-05-08T15:52:58.246 INFO:teuthology.orchestra.run.smithi172.stderr: 8 May 15:52:58 ntpd[17263]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-05-08T15:52:58.246 INFO:teuthology.orchestra.run.smithi172.stderr: 8 May 15:52:58 ntpd[17263]: can't open /var/log/ntpstats/rawstats.20240508: Permission denied 2024-05-08T15:52:58.246 INFO:teuthology.orchestra.run.smithi172.stderr: 8 May 15:52:58 ntpd[17263]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-05-08T15:52:58.246 INFO:teuthology.orchestra.run.smithi172.stderr: 8 May 15:52:58 ntpd[17263]: can't open /var/log/ntpstats/peerstats.20240508: Permission denied 2024-05-08T15:52:58.246 INFO:teuthology.orchestra.run.smithi172.stdout: 8 May 15:52:58 ntpd[17263]: ntpd: time slew -0.001146 s 2024-05-08T15:52:58.247 INFO:teuthology.orchestra.run.smithi172.stdout:ntpd: time slew -0.001146s 2024-05-08T15:52:58.247 INFO:teuthology.orchestra.run.smithi172.stderr: 8 May 15:52:58 ntpd[17263]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2024-05-08T15:52:58.247 INFO:teuthology.orchestra.run.smithi172.stderr: 8 May 15:52:58 ntpd[17263]: can't open /var/log/ntpstats/loopstats.20240508: Permission denied 2024-05-08T15:52:58.360 INFO:teuthology.orchestra.run.smithi172.stdout: remote refid st t when poll reach delay offset jitter 2024-05-08T15:52:58.361 INFO:teuthology.orchestra.run.smithi172.stdout:============================================================================== 2024-05-08T15:52:58.361 INFO:teuthology.orchestra.run.smithi172.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-05-08T15:52:58.361 INFO:teuthology.orchestra.run.smithi172.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-05-08T15:52:58.361 INFO:teuthology.orchestra.run.smithi172.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-05-08T15:52:58.361 INFO:teuthology.orchestra.run.smithi172.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-05-08T15:52:58.361 INFO:teuthology.run_tasks:Running task install... 2024-05-08T15:52:58.369 DEBUG:teuthology.task.install:project ceph 2024-05-08T15:52:58.370 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '95a4ad3c90f2f12f920e87725ad8d9002f948722'}} 2024-05-08T15:52:58.370 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': '95a4ad3c90f2f12f920e87725ad8d9002f948722'} 2024-05-08T15:52:58.370 INFO:teuthology.task.install:Using flavor: default 2024-05-08T15:52:58.376 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-05-08T15:52:58.376 INFO:teuthology.task.install:extra packages: [] 2024-05-08T15:52:58.376 DEBUG:teuthology.orchestra.run.smithi172:> sudo apt-key list | grep Ceph 2024-05-08T15:52:58.467 INFO:teuthology.orchestra.run.smithi172.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2024-05-08T15:52:58.495 INFO:teuthology.orchestra.run.smithi172.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2024-05-08T15:52:58.495 INFO:teuthology.orchestra.run.smithi172.stdout:uid [ unknown] Ceph.com (release key) 2024-05-08T15:52:58.496 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2024-05-08T15:52:58.496 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=95a4ad3c90f2f12f920e87725ad8d9002f948722 2024-05-08T15:52:58.725 INFO:teuthology.task.install.deb:Pulling from https://4.chacra.ceph.com/r/ceph/wip-lflores-testing-2-2024-05-07-1606-squid/95a4ad3c90f2f12f920e87725ad8d9002f948722/ubuntu/jammy/flavors/default/ 2024-05-08T15:52:58.725 INFO:teuthology.task.install.deb:Package version is 19.0.0-2517-g95a4ad3c-1jammy 2024-05-08T15:52:58.896 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2024-05-08T15:52:58.896 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2024-05-08T15:52:58.910 DEBUG:teuthology.orchestra.run.smithi172:> sudo apt-get update 2024-05-08T15:52:59.081 INFO:teuthology.orchestra.run.smithi172.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-05-08T15:52:59.093 INFO:teuthology.orchestra.run.smithi172.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-05-08T15:52:59.114 INFO:teuthology.orchestra.run.smithi172.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-05-08T15:52:59.186 INFO:teuthology.orchestra.run.smithi172.stdout:Ign:4 https://4.chacra.ceph.com/r/ceph/wip-lflores-testing-2-2024-05-07-1606-squid/95a4ad3c90f2f12f920e87725ad8d9002f948722/ubuntu/jammy/flavors/default jammy InRelease 2024-05-08T15:52:59.194 INFO:teuthology.orchestra.run.smithi172.stdout:Hit:5 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-05-08T15:52:59.223 INFO:teuthology.orchestra.run.smithi172.stdout:Get:6 https://4.chacra.ceph.com/r/ceph/wip-lflores-testing-2-2024-05-07-1606-squid/95a4ad3c90f2f12f920e87725ad8d9002f948722/ubuntu/jammy/flavors/default jammy Release [7,662 B] 2024-05-08T15:52:59.497 INFO:teuthology.orchestra.run.smithi172.stdout:Ign:7 https://4.chacra.ceph.com/r/ceph/wip-lflores-testing-2-2024-05-07-1606-squid/95a4ad3c90f2f12f920e87725ad8d9002f948722/ubuntu/jammy/flavors/default jammy Release.gpg 2024-05-08T15:52:59.533 INFO:teuthology.orchestra.run.smithi172.stdout:Get:8 https://4.chacra.ceph.com/r/ceph/wip-lflores-testing-2-2024-05-07-1606-squid/95a4ad3c90f2f12f920e87725ad8d9002f948722/ubuntu/jammy/flavors/default jammy/main i386 Packages [3,410 B] 2024-05-08T15:52:59.615 INFO:teuthology.orchestra.run.smithi172.stdout:Get:9 https://4.chacra.ceph.com/r/ceph/wip-lflores-testing-2-2024-05-07-1606-squid/95a4ad3c90f2f12f920e87725ad8d9002f948722/ubuntu/jammy/flavors/default jammy/main amd64 Packages [17.9 kB] 2024-05-08T15:52:59.706 INFO:teuthology.orchestra.run.smithi172.stdout:Fetched 28.9 kB in 1s (47.8 kB/s) 2024-05-08T15:53:00.742 INFO:teuthology.orchestra.run.smithi172.stdout:Reading package lists... 2024-05-08T15:53:00.759 DEBUG:teuthology.orchestra.run.smithi172:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=19.0.0-2517-g95a4ad3c-1jammy cephadm=19.0.0-2517-g95a4ad3c-1jammy ceph-mds=19.0.0-2517-g95a4ad3c-1jammy ceph-mgr=19.0.0-2517-g95a4ad3c-1jammy ceph-common=19.0.0-2517-g95a4ad3c-1jammy ceph-fuse=19.0.0-2517-g95a4ad3c-1jammy ceph-test=19.0.0-2517-g95a4ad3c-1jammy ceph-volume=19.0.0-2517-g95a4ad3c-1jammy radosgw=19.0.0-2517-g95a4ad3c-1jammy python3-rados=19.0.0-2517-g95a4ad3c-1jammy python3-rgw=19.0.0-2517-g95a4ad3c-1jammy python3-cephfs=19.0.0-2517-g95a4ad3c-1jammy python3-rbd=19.0.0-2517-g95a4ad3c-1jammy libcephfs2=19.0.0-2517-g95a4ad3c-1jammy libcephfs-dev=19.0.0-2517-g95a4ad3c-1jammy librados2=19.0.0-2517-g95a4ad3c-1jammy librbd1=19.0.0-2517-g95a4ad3c-1jammy rbd-fuse=19.0.0-2517-g95a4ad3c-1jammy 2024-05-08T15:53:00.822 INFO:teuthology.orchestra.run.smithi172.stdout:Reading package lists... 2024-05-08T15:53:01.030 INFO:teuthology.orchestra.run.smithi172.stdout:Building dependency tree... 2024-05-08T15:53:01.030 INFO:teuthology.orchestra.run.smithi172.stdout:Reading state information... 2024-05-08T15:53:01.222 INFO:teuthology.orchestra.run.smithi172.stdout:The following packages were automatically installed and are no longer required: 2024-05-08T15:53:01.222 INFO:teuthology.orchestra.run.smithi172.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-05-08T15:53:01.222 INFO:teuthology.orchestra.run.smithi172.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-05-08T15:53:01.222 INFO:teuthology.orchestra.run.smithi172.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-05-08T15:53:01.223 INFO:teuthology.orchestra.run.smithi172.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-05-08T15:53:01.223 INFO:teuthology.orchestra.run.smithi172.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-05-08T15:53:01.223 INFO:teuthology.orchestra.run.smithi172.stdout:Use 'sudo apt autoremove' to remove them. 2024-05-08T15:53:01.223 INFO:teuthology.orchestra.run.smithi172.stdout:The following additional packages will be installed: 2024-05-08T15:53:01.223 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2024-05-08T15:53:01.225 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2024-05-08T15:53:01.225 INFO:teuthology.orchestra.run.smithi172.stdout: libdouble-conversion3 libjq1 liblttng-ust1 liblua5.3-dev liboath0 libonig5 2024-05-08T15:53:01.225 INFO:teuthology.orchestra.run.smithi172.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 libradosstriper1 2024-05-08T15:53:01.225 INFO:teuthology.orchestra.run.smithi172.stdout: librdkafka1 libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 2024-05-08T15:53:01.225 INFO:teuthology.orchestra.run.smithi172.stdout: lua-any lua-sec lua-socket lua5.1 luarocks nvme-cli pkg-config 2024-05-08T15:53:01.225 INFO:teuthology.orchestra.run.smithi172.stdout: python-asyncssh-doc python-babel-localedata python-pastedeploy-tpl 2024-05-08T15:53:01.225 INFO:teuthology.orchestra.run.smithi172.stdout: python3-asyncssh python3-babel python3-bcrypt python3-cachetools 2024-05-08T15:53:01.225 INFO:teuthology.orchestra.run.smithi172.stdout: python3-ceph-argparse python3-ceph-common python3-certifi python3-cheroot 2024-05-08T15:53:01.225 INFO:teuthology.orchestra.run.smithi172.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-05-08T15:53:01.226 INFO:teuthology.orchestra.run.smithi172.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-05-08T15:53:01.226 INFO:teuthology.orchestra.run.smithi172.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-05-08T15:53:01.226 INFO:teuthology.orchestra.run.smithi172.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-05-08T15:53:01.226 INFO:teuthology.orchestra.run.smithi172.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-05-08T15:53:01.226 INFO:teuthology.orchestra.run.smithi172.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-05-08T15:53:01.226 INFO:teuthology.orchestra.run.smithi172.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-05-08T15:53:01.226 INFO:teuthology.orchestra.run.smithi172.stdout: python3-pygments python3-pyinotify python3-pytest python3-repoze.lru 2024-05-08T15:53:01.226 INFO:teuthology.orchestra.run.smithi172.stdout: python3-requests python3-requests-oauthlib python3-routes python3-rsa 2024-05-08T15:53:01.227 INFO:teuthology.orchestra.run.smithi172.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2024-05-08T15:53:01.227 INFO:teuthology.orchestra.run.smithi172.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2024-05-08T15:53:01.227 INFO:teuthology.orchestra.run.smithi172.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2024-05-08T15:53:01.228 INFO:teuthology.orchestra.run.smithi172.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2024-05-08T15:53:01.228 INFO:teuthology.orchestra.run.smithi172.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n socat unzip 2024-05-08T15:53:01.228 INFO:teuthology.orchestra.run.smithi172.stdout: xmlstarlet zip 2024-05-08T15:53:01.228 INFO:teuthology.orchestra.run.smithi172.stdout:Suggested packages: 2024-05-08T15:53:01.228 INFO:teuthology.orchestra.run.smithi172.stdout: python3-influxdb readline-doc python-jinja2-doc python3-beaker 2024-05-08T15:53:01.228 INFO:teuthology.orchestra.run.smithi172.stdout: python-mako-doc python-natsort-doc python-openssl-doc python3-openssl-dbg 2024-05-08T15:53:01.228 INFO:teuthology.orchestra.run.smithi172.stdout: httpd-wsgi libapache2-mod-python libapache2-mod-scgi libjs-mochikit 2024-05-08T15:53:01.229 INFO:teuthology.orchestra.run.smithi172.stdout: python-pecan-doc python-psutil-doc subversion python-pygments-doc 2024-05-08T15:53:01.229 INFO:teuthology.orchestra.run.smithi172.stdout: python-pyinotify-doc python3-socks python-requests-doc python3-dap 2024-05-08T15:53:01.229 INFO:teuthology.orchestra.run.smithi172.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2024-05-08T15:53:01.229 INFO:teuthology.orchestra.run.smithi172.stdout: python-webtest-doc python-werkzeug-doc python3-watchdog 2024-05-08T15:53:01.229 INFO:teuthology.orchestra.run.smithi172.stdout:Recommended packages: 2024-05-08T15:53:01.229 INFO:teuthology.orchestra.run.smithi172.stdout: btrfs-tools 2024-05-08T15:53:01.324 INFO:teuthology.orchestra.run.smithi172.stdout:The following NEW packages will be installed: 2024-05-08T15:53:01.324 INFO:teuthology.orchestra.run.smithi172.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2024-05-08T15:53:01.324 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2024-05-08T15:53:01.324 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm jq 2024-05-08T15:53:01.324 INFO:teuthology.orchestra.run.smithi172.stdout: libcephfs-dev libcephfs2 libdouble-conversion3 libjq1 liblttng-ust1 2024-05-08T15:53:01.324 INFO:teuthology.orchestra.run.smithi172.stdout: liblua5.3-dev liboath0 libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 2024-05-08T15:53:01.324 INFO:teuthology.orchestra.run.smithi172.stdout: libqt5network5 librados2 libradosstriper1 librbd1 librdkafka1 2024-05-08T15:53:01.326 INFO:teuthology.orchestra.run.smithi172.stdout: libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 lua-any lua-sec 2024-05-08T15:53:01.326 INFO:teuthology.orchestra.run.smithi172.stdout: lua-socket lua5.1 luarocks nvme-cli pkg-config python-asyncssh-doc 2024-05-08T15:53:01.326 INFO:teuthology.orchestra.run.smithi172.stdout: python-babel-localedata python-pastedeploy-tpl python3-asyncssh 2024-05-08T15:53:01.326 INFO:teuthology.orchestra.run.smithi172.stdout: python3-babel python3-bcrypt python3-cachetools python3-ceph-argparse 2024-05-08T15:53:01.326 INFO:teuthology.orchestra.run.smithi172.stdout: python3-ceph-common python3-cephfs python3-certifi python3-cheroot 2024-05-08T15:53:01.326 INFO:teuthology.orchestra.run.smithi172.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-05-08T15:53:01.326 INFO:teuthology.orchestra.run.smithi172.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-05-08T15:53:01.326 INFO:teuthology.orchestra.run.smithi172.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-05-08T15:53:01.326 INFO:teuthology.orchestra.run.smithi172.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-05-08T15:53:01.326 INFO:teuthology.orchestra.run.smithi172.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-05-08T15:53:01.326 INFO:teuthology.orchestra.run.smithi172.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-05-08T15:53:01.326 INFO:teuthology.orchestra.run.smithi172.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-05-08T15:53:01.327 INFO:teuthology.orchestra.run.smithi172.stdout: python3-pygments python3-pyinotify python3-pytest python3-rados python3-rbd 2024-05-08T15:53:01.327 INFO:teuthology.orchestra.run.smithi172.stdout: python3-repoze.lru python3-requests python3-requests-oauthlib python3-rgw 2024-05-08T15:53:01.327 INFO:teuthology.orchestra.run.smithi172.stdout: python3-routes python3-rsa python3-simplegeneric python3-simplejson 2024-05-08T15:53:01.327 INFO:teuthology.orchestra.run.smithi172.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib python3-tempita 2024-05-08T15:53:01.327 INFO:teuthology.orchestra.run.smithi172.stdout: python3-tempora python3-threadpoolctl python3-toml python3-urllib3 2024-05-08T15:53:01.327 INFO:teuthology.orchestra.run.smithi172.stdout: python3-waitress python3-wcwidth python3-webob python3-websocket 2024-05-08T15:53:01.327 INFO:teuthology.orchestra.run.smithi172.stdout: python3-webtest python3-werkzeug python3-zc.lockfile qttranslations5-l10n 2024-05-08T15:53:01.327 INFO:teuthology.orchestra.run.smithi172.stdout: radosgw rbd-fuse socat unzip xmlstarlet zip 2024-05-08T15:53:01.511 INFO:teuthology.orchestra.run.smithi172.stdout:0 upgraded, 118 newly installed, 0 to remove and 303 not upgraded. 2024-05-08T15:53:01.511 INFO:teuthology.orchestra.run.smithi172.stdout:Need to get 180 MB of archives. 2024-05-08T15:53:01.511 INFO:teuthology.orchestra.run.smithi172.stdout:After this operation, 822 MB of additional disk space will be used. 2024-05-08T15:53:01.511 INFO:teuthology.orchestra.run.smithi172.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2024-05-08T15:53:01.539 INFO:teuthology.orchestra.run.smithi172.stdout:Get:2 https://4.chacra.ceph.com/r/ceph/wip-lflores-testing-2-2024-05-07-1606-squid/95a4ad3c90f2f12f920e87725ad8d9002f948722/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 19.0.0-2517-g95a4ad3c-1jammy [3,546 kB] 2024-05-08T15:53:01.853 INFO:teuthology.orchestra.run.smithi172.stdout:Get:3 https://4.chacra.ceph.com/r/ceph/wip-lflores-testing-2-2024-05-07-1606-squid/95a4ad3c90f2f12f920e87725ad8d9002f948722/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 19.0.0-2517-g95a4ad3c-1jammy [3,216 kB] 2024-05-08T15:53:01.933 INFO:teuthology.orchestra.run.smithi172.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2024-05-08T15:53:01.946 INFO:teuthology.orchestra.run.smithi172.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2024-05-08T15:53:01.950 INFO:teuthology.orchestra.run.smithi172.stdout:Get:6 https://4.chacra.ceph.com/r/ceph/wip-lflores-testing-2-2024-05-07-1606-squid/95a4ad3c90f2f12f920e87725ad8d9002f948722/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 19.0.0-2517-g95a4ad3c-1jammy [711 kB] 2024-05-08T15:53:01.974 INFO:teuthology.orchestra.run.smithi172.stdout:Get:7 https://4.chacra.ceph.com/r/ceph/wip-lflores-testing-2-2024-05-07-1606-squid/95a4ad3c90f2f12f920e87725ad8d9002f948722/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 19.0.0-2517-g95a4ad3c-1jammy [356 kB] 2024-05-08T15:53:01.985 INFO:teuthology.orchestra.run.smithi172.stdout:Get:8 https://4.chacra.ceph.com/r/ceph/wip-lflores-testing-2-2024-05-07-1606-squid/95a4ad3c90f2f12f920e87725ad8d9002f948722/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 19.0.0-2517-g95a4ad3c-1jammy [32.8 kB] 2024-05-08T15:53:01.986 INFO:teuthology.orchestra.run.smithi172.stdout:Get:9 https://4.chacra.ceph.com/r/ceph/wip-lflores-testing-2-2024-05-07-1606-squid/95a4ad3c90f2f12f920e87725ad8d9002f948722/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 19.0.0-2517-g95a4ad3c-1jammy [180 kB] 2024-05-08T15:53:01.991 INFO:teuthology.orchestra.run.smithi172.stdout:Get:10 https://4.chacra.ceph.com/r/ceph/wip-lflores-testing-2-2024-05-07-1606-squid/95a4ad3c90f2f12f920e87725ad8d9002f948722/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 19.0.0-2517-g95a4ad3c-1jammy [67.1 kB] 2024-05-08T15:53:01.994 INFO:teuthology.orchestra.run.smithi172.stdout:Get:11 https://4.chacra.ceph.com/r/ceph/wip-lflores-testing-2-2024-05-07-1606-squid/95a4ad3c90f2f12f920e87725ad8d9002f948722/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 19.0.0-2517-g95a4ad3c-1jammy [330 kB] 2024-05-08T15:53:02.005 INFO:teuthology.orchestra.run.smithi172.stdout:Get:12 https://4.chacra.ceph.com/r/ceph/wip-lflores-testing-2-2024-05-07-1606-squid/95a4ad3c90f2f12f920e87725ad8d9002f948722/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 19.0.0-2517-g95a4ad3c-1jammy [6,779 kB] 2024-05-08T15:53:02.027 INFO:teuthology.orchestra.run.smithi172.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2,006 kB] 2024-05-08T15:53:02.229 INFO:teuthology.orchestra.run.smithi172.stdout:Get:14 https://4.chacra.ceph.com/r/ceph/wip-lflores-testing-2-2024-05-07-1606-squid/95a4ad3c90f2f12f920e87725ad8d9002f948722/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 19.0.0-2517-g95a4ad3c-1jammy [112 kB] 2024-05-08T15:53:02.232 INFO:teuthology.orchestra.run.smithi172.stdout:Get:15 https://4.chacra.ceph.com/r/ceph/wip-lflores-testing-2-2024-05-07-1606-squid/95a4ad3c90f2f12f920e87725ad8d9002f948722/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 19.0.0-2517-g95a4ad3c-1jammy [466 kB] 2024-05-08T15:53:02.245 INFO:teuthology.orchestra.run.smithi172.stdout:Get:16 https://4.chacra.ceph.com/r/ceph/wip-lflores-testing-2-2024-05-07-1606-squid/95a4ad3c90f2f12f920e87725ad8d9002f948722/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 19.0.0-2517-g95a4ad3c-1jammy [25.6 MB] 2024-05-08T15:53:02.257 INFO:teuthology.orchestra.run.smithi172.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2024-05-08T15:53:02.269 INFO:teuthology.orchestra.run.smithi172.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2024-05-08T15:53:02.299 INFO:teuthology.orchestra.run.smithi172.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2024-05-08T15:53:02.309 INFO:teuthology.orchestra.run.smithi172.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2024-05-08T15:53:02.310 INFO:teuthology.orchestra.run.smithi172.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2024-05-08T15:53:02.311 INFO:teuthology.orchestra.run.smithi172.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2024-05-08T15:53:02.330 INFO:teuthology.orchestra.run.smithi172.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy/main amd64 libreadline-dev amd64 8.1.2-1 [166 kB] 2024-05-08T15:53:02.498 INFO:teuthology.orchestra.run.smithi172.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblua5.3-dev amd64 5.3.6-1build1 [167 kB] 2024-05-08T15:53:02.907 INFO:teuthology.orchestra.run.smithi172.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua5.1 amd64 5.1.5-8.1build4 [94.6 kB] 2024-05-08T15:53:02.941 INFO:teuthology.orchestra.run.smithi172.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-any all 27ubuntu1 [5,034 B] 2024-05-08T15:53:02.943 INFO:teuthology.orchestra.run.smithi172.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy/main amd64 zip amd64 3.0-12build2 [176 kB] 2024-05-08T15:53:03.023 INFO:teuthology.orchestra.run.smithi172.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 unzip amd64 6.0-26ubuntu3.2 [175 kB] 2024-05-08T15:53:03.074 INFO:teuthology.orchestra.run.smithi172.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/universe amd64 luarocks all 3.8.0+dfsg1-1 [140 kB] 2024-05-08T15:53:03.095 INFO:teuthology.orchestra.run.smithi172.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy/main amd64 liboath0 amd64 2.6.7-3build1 [41.1 kB] 2024-05-08T15:53:03.102 INFO:teuthology.orchestra.run.smithi172.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9,030 B] 2024-05-08T15:53:03.103 INFO:teuthology.orchestra.run.smithi172.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2024-05-08T15:53:03.158 INFO:teuthology.orchestra.run.smithi172.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6,452 B] 2024-05-08T15:53:03.159 INFO:teuthology.orchestra.run.smithi172.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8,716 B] 2024-05-08T15:53:03.249 INFO:teuthology.orchestra.run.smithi172.stdout:Get:35 https://4.chacra.ceph.com/r/ceph/wip-lflores-testing-2-2024-05-07-1606-squid/95a4ad3c90f2f12f920e87725ad8d9002f948722/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 19.0.0-2517-g95a4ad3c-1jammy [5,085 kB] 2024-05-08T15:53:03.325 INFO:teuthology.orchestra.run.smithi172.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2024-05-08T15:53:03.330 INFO:teuthology.orchestra.run.smithi172.stdout:Get:37 https://4.chacra.ceph.com/r/ceph/wip-lflores-testing-2-2024-05-07-1606-squid/95a4ad3c90f2f12f920e87725ad8d9002f948722/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 19.0.0-2517-g95a4ad3c-1jammy [244 kB] 2024-05-08T15:53:03.333 INFO:teuthology.orchestra.run.smithi172.stdout:Get:38 https://4.chacra.ceph.com/r/ceph/wip-lflores-testing-2-2024-05-07-1606-squid/95a4ad3c90f2f12f920e87725ad8d9002f948722/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 19.0.0-2517-g95a4ad3c-1jammy [124 kB] 2024-05-08T15:53:03.335 INFO:teuthology.orchestra.run.smithi172.stdout:Get:39 https://4.chacra.ceph.com/r/ceph/wip-lflores-testing-2-2024-05-07-1606-squid/95a4ad3c90f2f12f920e87725ad8d9002f948722/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 19.0.0-2517-g95a4ad3c-1jammy [1,526 kB] 2024-05-08T15:53:03.378 INFO:teuthology.orchestra.run.smithi172.stdout:Get:40 https://4.chacra.ceph.com/r/ceph/wip-lflores-testing-2-2024-05-07-1606-squid/95a4ad3c90f2f12f920e87725ad8d9002f948722/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 19.0.0-2517-g95a4ad3c-1jammy [6,246 kB] 2024-05-08T15:53:03.408 INFO:teuthology.orchestra.run.smithi172.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2024-05-08T15:53:03.436 INFO:teuthology.orchestra.run.smithi172.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7,240 B] 2024-05-08T15:53:03.453 INFO:teuthology.orchestra.run.smithi172.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8,980 B] 2024-05-08T15:53:03.498 INFO:teuthology.orchestra.run.smithi172.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2024-05-08T15:53:03.584 INFO:teuthology.orchestra.run.smithi172.stdout:Get:45 https://4.chacra.ceph.com/r/ceph/wip-lflores-testing-2-2024-05-07-1606-squid/95a4ad3c90f2f12f920e87725ad8d9002f948722/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 19.0.0-2517-g95a4ad3c-1jammy [22.7 MB] 2024-05-08T15:53:03.712 INFO:teuthology.orchestra.run.smithi172.stdout:Get:46 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2024-05-08T15:53:03.727 INFO:teuthology.orchestra.run.smithi172.stdout:Get:47 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-logutils all 0.3.3-8 [17.6 kB] 2024-05-08T15:53:03.733 INFO:teuthology.orchestra.run.smithi172.stdout:Get:48 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-markupsafe amd64 2.0.1-2build1 [12.7 kB] 2024-05-08T15:53:03.739 INFO:teuthology.orchestra.run.smithi172.stdout:Get:49 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-mako all 1.1.3+ds1-2ubuntu0.1 [60.5 kB] 2024-05-08T15:53:03.760 INFO:teuthology.orchestra.run.smithi172.stdout:Get:50 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2024-05-08T15:53:03.764 INFO:teuthology.orchestra.run.smithi172.stdout:Get:51 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-singledispatch all 3.4.0.3-3 [7,320 B] 2024-05-08T15:53:03.930 INFO:teuthology.orchestra.run.smithi172.stdout:Get:52 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webob all 1:1.8.6-1.1 [85.2 kB] 2024-05-08T15:53:04.262 INFO:teuthology.orchestra.run.smithi172.stdout:Get:53 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-waitress all 1.4.4-1.1ubuntu1 [47.3 kB] 2024-05-08T15:53:04.294 INFO:teuthology.orchestra.run.smithi172.stdout:Get:54 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempita all 0.5.2-6ubuntu1 [15.1 kB] 2024-05-08T15:53:04.304 INFO:teuthology.orchestra.run.smithi172.stdout:Get:55 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-paste all 3.5.0+dfsg1-1 [456 kB] 2024-05-08T15:53:04.441 INFO:teuthology.orchestra.run.smithi172.stdout:Get:56 https://4.chacra.ceph.com/r/ceph/wip-lflores-testing-2-2024-05-07-1606-squid/95a4ad3c90f2f12f920e87725ad8d9002f948722/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 19.0.0-2517-g95a4ad3c-1jammy [14.1 kB] 2024-05-08T15:53:04.441 INFO:teuthology.orchestra.run.smithi172.stdout:Get:57 https://4.chacra.ceph.com/r/ceph/wip-lflores-testing-2-2024-05-07-1606-squid/95a4ad3c90f2f12f920e87725ad8d9002f948722/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 19.0.0-2517-g95a4ad3c-1jammy [869 kB] 2024-05-08T15:53:04.459 INFO:teuthology.orchestra.run.smithi172.stdout:Get:58 https://4.chacra.ceph.com/r/ceph/wip-lflores-testing-2-2024-05-07-1606-squid/95a4ad3c90f2f12f920e87725ad8d9002f948722/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 19.0.0-2517-g95a4ad3c-1jammy [2,437 kB] 2024-05-08T15:53:04.502 INFO:teuthology.orchestra.run.smithi172.stdout:Get:59 https://4.chacra.ceph.com/r/ceph/wip-lflores-testing-2-2024-05-07-1606-squid/95a4ad3c90f2f12f920e87725ad8d9002f948722/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 19.0.0-2517-g95a4ad3c-1jammy [791 kB] 2024-05-08T15:53:04.506 INFO:teuthology.orchestra.run.smithi172.stdout:Get:60 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-pastedeploy-tpl all 2.1.1-1 [4,892 B] 2024-05-08T15:53:04.507 INFO:teuthology.orchestra.run.smithi172.stdout:Get:61 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastedeploy all 2.1.1-1 [26.6 kB] 2024-05-08T15:53:04.511 INFO:teuthology.orchestra.run.smithi172.stdout:Get:62 https://4.chacra.ceph.com/r/ceph/wip-lflores-testing-2-2024-05-07-1606-squid/95a4ad3c90f2f12f920e87725ad8d9002f948722/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 19.0.0-2517-g95a4ad3c-1jammy [148 kB] 2024-05-08T15:53:04.513 INFO:teuthology.orchestra.run.smithi172.stdout:Get:63 https://4.chacra.ceph.com/r/ceph/wip-lflores-testing-2-2024-05-07-1606-squid/95a4ad3c90f2f12f920e87725ad8d9002f948722/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 19.0.0-2517-g95a4ad3c-1jammy [2,331 kB] 2024-05-08T15:53:04.513 INFO:teuthology.orchestra.run.smithi172.stdout:Get:64 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webtest all 2.0.35-1 [28.5 kB] 2024-05-08T15:53:04.519 INFO:teuthology.orchestra.run.smithi172.stdout:Get:65 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pecan all 1.3.3-4ubuntu2 [87.3 kB] 2024-05-08T15:53:04.534 INFO:teuthology.orchestra.run.smithi172.stdout:Get:66 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-certifi all 2020.6.20-1 [150 kB] 2024-05-08T15:53:04.555 INFO:teuthology.orchestra.run.smithi172.stdout:Get:67 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-idna all 3.3-1 [49.3 kB] 2024-05-08T15:53:04.561 INFO:teuthology.orchestra.run.smithi172.stdout:Get:68 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-urllib3 all 1.26.5-1~exp1ubuntu0.1 [98.2 kB] 2024-05-08T15:53:04.568 INFO:teuthology.orchestra.run.smithi172.stdout:Get:69 https://4.chacra.ceph.com/r/ceph/wip-lflores-testing-2-2024-05-07-1606-squid/95a4ad3c90f2f12f920e87725ad8d9002f948722/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 19.0.0-2517-g95a4ad3c-1jammy [8,625 kB] 2024-05-08T15:53:04.736 INFO:teuthology.orchestra.run.smithi172.stdout:Get:70 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-requests all 2.25.1+dfsg-2ubuntu0.1 [48.8 kB] 2024-05-08T15:53:04.852 INFO:teuthology.orchestra.run.smithi172.stdout:Get:71 https://4.chacra.ceph.com/r/ceph/wip-lflores-testing-2-2024-05-07-1606-squid/95a4ad3c90f2f12f920e87725ad8d9002f948722/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 19.0.0-2517-g95a4ad3c-1jammy [14.2 kB] 2024-05-08T15:53:04.853 INFO:teuthology.orchestra.run.smithi172.stdout:Get:72 https://4.chacra.ceph.com/r/ceph/wip-lflores-testing-2-2024-05-07-1606-squid/95a4ad3c90f2f12f920e87725ad8d9002f948722/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 19.0.0-2517-g95a4ad3c-1jammy [50.7 MB] 2024-05-08T15:53:04.990 INFO:teuthology.orchestra.run.smithi172.stdout:Get:73 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-werkzeug all 2.0.2+dfsg1-1ubuntu0.22.04.1 [181 kB] 2024-05-08T15:53:05.166 INFO:teuthology.orchestra.run.smithi172.stdout:Get:74 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-bcrypt amd64 3.2.0-1build1 [32.7 kB] 2024-05-08T15:53:05.180 INFO:teuthology.orchestra.run.smithi172.stdout:Get:75 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-openssl all 21.0.0-1 [45.2 kB] 2024-05-08T15:53:05.200 INFO:teuthology.orchestra.run.smithi172.stdout:Get:76 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-asyncssh all 2.5.0-1 [185 kB] 2024-05-08T15:53:05.264 INFO:teuthology.orchestra.run.smithi172.stdout:Get:77 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-babel-localedata all 2.8.0+dfsg.1-7 [4,982 kB] 2024-05-08T15:53:05.563 INFO:teuthology.orchestra.run.smithi172.stdout:Get:78 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-babel all 2.8.0+dfsg.1-7 [85.1 kB] 2024-05-08T15:53:05.566 INFO:teuthology.orchestra.run.smithi172.stdout:Get:79 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-jinja2 all 3.0.3-1ubuntu0.1 [108 kB] 2024-05-08T15:53:05.567 INFO:teuthology.orchestra.run.smithi172.stdout:Get:80 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2024-05-08T15:53:05.568 INFO:teuthology.orchestra.run.smithi172.stdout:Get:81 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2024-05-08T15:53:05.570 INFO:teuthology.orchestra.run.smithi172.stdout:Get:82 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2,058 kB] 2024-05-08T15:53:05.779 INFO:teuthology.orchestra.run.smithi172.stdout:Get:83 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2024-05-08T15:53:06.217 INFO:teuthology.orchestra.run.smithi172.stdout:Get:84 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2024-05-08T15:53:06.223 INFO:teuthology.orchestra.run.smithi172.stdout:Get:85 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1,829 kB] 2024-05-08T15:53:06.514 INFO:teuthology.orchestra.run.smithi172.stdout:Get:86 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9,722 B] 2024-05-08T15:53:06.514 INFO:teuthology.orchestra.run.smithi172.stdout:Get:87 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1 all 0.4.8-1 [50.9 kB] 2024-05-08T15:53:06.516 INFO:teuthology.orchestra.run.smithi172.stdout:Get:88 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1-modules all 0.2.1-1 [33.0 kB] 2024-05-08T15:53:06.572 INFO:teuthology.orchestra.run.smithi172.stdout:Get:89 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2024-05-08T15:53:06.573 INFO:teuthology.orchestra.run.smithi172.stdout:Get:90 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2024-05-08T15:53:06.617 INFO:teuthology.orchestra.run.smithi172.stdout:Get:91 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2024-05-08T15:53:06.687 INFO:teuthology.orchestra.run.smithi172.stdout:Get:92 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2024-05-08T15:53:06.689 INFO:teuthology.orchestra.run.smithi172.stdout:Get:93 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2024-05-08T15:53:06.788 INFO:teuthology.orchestra.run.smithi172.stdout:Get:94 https://4.chacra.ceph.com/r/ceph/wip-lflores-testing-2-2024-05-07-1606-squid/95a4ad3c90f2f12f920e87725ad8d9002f948722/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 19.0.0-2517-g95a4ad3c-1jammy [121 kB] 2024-05-08T15:53:06.790 INFO:teuthology.orchestra.run.smithi172.stdout:Get:95 https://4.chacra.ceph.com/r/ceph/wip-lflores-testing-2-2024-05-07-1606-squid/95a4ad3c90f2f12f920e87725ad8d9002f948722/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 19.0.0-2517-g95a4ad3c-1jammy [38.9 kB] 2024-05-08T15:53:06.791 INFO:teuthology.orchestra.run.smithi172.stdout:Get:96 https://4.chacra.ceph.com/r/ceph/wip-lflores-testing-2-2024-05-07-1606-squid/95a4ad3c90f2f12f920e87725ad8d9002f948722/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 19.0.0-2517-g95a4ad3c-1jammy [13.4 MB] 2024-05-08T15:53:06.949 INFO:teuthology.orchestra.run.smithi172.stdout:Get:97 http://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2024-05-08T15:53:07.142 INFO:teuthology.orchestra.run.smithi172.stdout:Get:98 https://4.chacra.ceph.com/r/ceph/wip-lflores-testing-2-2024-05-07-1606-squid/95a4ad3c90f2f12f920e87725ad8d9002f948722/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 19.0.0-2517-g95a4ad3c-1jammy [91.6 kB] 2024-05-08T15:53:07.363 INFO:teuthology.orchestra.run.smithi172.stdout:Get:99 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjq1 amd64 1.6-2.1ubuntu3 [133 kB] 2024-05-08T15:53:07.411 INFO:teuthology.orchestra.run.smithi172.stdout:Get:100 http://archive.ubuntu.com/ubuntu jammy/main amd64 jq amd64 1.6-2.1ubuntu3 [52.5 kB] 2024-05-08T15:53:07.430 INFO:teuthology.orchestra.run.smithi172.stdout:Get:101 http://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2024-05-08T15:53:07.541 INFO:teuthology.orchestra.run.smithi172.stdout:Get:102 http://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2024-05-08T15:53:07.582 INFO:teuthology.orchestra.run.smithi172.stdout:Get:103 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-socket amd64 3.0~rc1+git+ac3201d-6 [78.9 kB] 2024-05-08T15:53:07.594 INFO:teuthology.orchestra.run.smithi172.stdout:Get:104 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-sec amd64 1.0.2-1 [37.6 kB] 2024-05-08T15:53:07.599 INFO:teuthology.orchestra.run.smithi172.stdout:Get:105 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.1 [474 kB] 2024-05-08T15:53:07.647 INFO:teuthology.orchestra.run.smithi172.stdout:Get:106 http://archive.ubuntu.com/ubuntu jammy/main amd64 pkg-config amd64 0.29.2-1ubuntu3 [48.2 kB] 2024-05-08T15:53:07.650 INFO:teuthology.orchestra.run.smithi172.stdout:Get:107 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python-asyncssh-doc all 2.5.0-1 [318 kB] 2024-05-08T15:53:07.671 INFO:teuthology.orchestra.run.smithi172.stdout:Get:108 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6,024 B] 2024-05-08T15:53:07.835 INFO:teuthology.orchestra.run.smithi172.stdout:Get:109 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2024-05-08T15:53:08.100 INFO:teuthology.orchestra.run.smithi172.stdout:Get:110 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2024-05-08T15:53:08.119 INFO:teuthology.orchestra.run.smithi172.stdout:Get:111 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2024-05-08T15:53:08.271 INFO:teuthology.orchestra.run.smithi172.stdout:Get:112 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2024-05-08T15:53:08.300 INFO:teuthology.orchestra.run.smithi172.stdout:Get:113 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pygments all 2.11.2+dfsg-2 [750 kB] 2024-05-08T15:53:08.457 INFO:teuthology.orchestra.run.smithi172.stdout:Get:114 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyinotify all 0.9.6-1.3 [24.8 kB] 2024-05-08T15:53:08.459 INFO:teuthology.orchestra.run.smithi172.stdout:Get:115 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2024-05-08T15:53:08.462 INFO:teuthology.orchestra.run.smithi172.stdout:Get:116 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2024-05-08T15:53:08.484 INFO:teuthology.orchestra.run.smithi172.stdout:Get:117 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2024-05-08T15:53:08.488 INFO:teuthology.orchestra.run.smithi172.stdout:Get:118 http://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1,983 kB] 2024-05-08T15:53:09.158 INFO:teuthology.orchestra.run.smithi172.stdout:Fetched 180 MB in 7s (24.7 MB/s) 2024-05-08T15:53:09.299 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2024-05-08T15:53:09.399 INFO:teuthology.orchestra.run.smithi172.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 165990 files and directories currently installed.) 2024-05-08T15:53:09.404 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../000-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2024-05-08T15:53:09.838 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-05-08T15:53:10.185 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2024-05-08T15:53:10.211 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../001-libdouble-conversion3_3.1.7-4_amd64.deb ... 2024-05-08T15:53:10.251 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2024-05-08T15:53:10.588 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2024-05-08T15:53:10.614 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../002-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2024-05-08T15:53:10.654 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-05-08T15:53:11.033 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package libqt5core5a:amd64. 2024-05-08T15:53:11.059 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../003-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-05-08T15:53:11.099 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-05-08T15:53:11.511 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2024-05-08T15:53:11.537 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../004-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-05-08T15:53:11.577 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-05-08T15:53:11.897 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package libqt5network5:amd64. 2024-05-08T15:53:11.924 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../005-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-05-08T15:53:11.956 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-05-08T15:53:12.300 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2024-05-08T15:53:12.327 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../006-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2024-05-08T15:53:12.367 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-05-08T15:53:12.695 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package librados2. 2024-05-08T15:53:12.721 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../007-librados2_19.0.0-2517-g95a4ad3c-1jammy_amd64.deb ... 2024-05-08T15:53:12.761 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking librados2 (19.0.0-2517-g95a4ad3c-1jammy) ... 2024-05-08T15:53:13.181 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package librbd1. 2024-05-08T15:53:13.196 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../008-librbd1_19.0.0-2517-g95a4ad3c-1jammy_amd64.deb ... 2024-05-08T15:53:13.253 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking librbd1 (19.0.0-2517-g95a4ad3c-1jammy) ... 2024-05-08T15:53:13.684 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package libcephfs2. 2024-05-08T15:53:13.710 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../009-libcephfs2_19.0.0-2517-g95a4ad3c-1jammy_amd64.deb ... 2024-05-08T15:53:13.750 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking libcephfs2 (19.0.0-2517-g95a4ad3c-1jammy) ... 2024-05-08T15:53:14.044 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-rados. 2024-05-08T15:53:14.060 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../010-python3-rados_19.0.0-2517-g95a4ad3c-1jammy_amd64.deb ... 2024-05-08T15:53:14.103 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-rados (19.0.0-2517-g95a4ad3c-1jammy) ... 2024-05-08T15:53:14.431 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-ceph-argparse. 2024-05-08T15:53:14.457 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../011-python3-ceph-argparse_19.0.0-2517-g95a4ad3c-1jammy_all.deb ... 2024-05-08T15:53:14.497 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-ceph-argparse (19.0.0-2517-g95a4ad3c-1jammy) ... 2024-05-08T15:53:14.733 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-cephfs. 2024-05-08T15:53:14.749 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../012-python3-cephfs_19.0.0-2517-g95a4ad3c-1jammy_amd64.deb ... 2024-05-08T15:53:14.783 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-cephfs (19.0.0-2517-g95a4ad3c-1jammy) ... 2024-05-08T15:53:15.111 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-ceph-common. 2024-05-08T15:53:15.137 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../013-python3-ceph-common_19.0.0-2517-g95a4ad3c-1jammy_all.deb ... 2024-05-08T15:53:15.177 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-ceph-common (19.0.0-2517-g95a4ad3c-1jammy) ... 2024-05-08T15:53:15.498 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-wcwidth. 2024-05-08T15:53:15.524 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../014-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2024-05-08T15:53:15.564 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2024-05-08T15:53:15.883 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-prettytable. 2024-05-08T15:53:15.910 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../015-python3-prettytable_2.5.0-2_all.deb ... 2024-05-08T15:53:15.950 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2024-05-08T15:53:16.202 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-rbd. 2024-05-08T15:53:16.218 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../016-python3-rbd_19.0.0-2517-g95a4ad3c-1jammy_amd64.deb ... 2024-05-08T15:53:16.252 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-rbd (19.0.0-2517-g95a4ad3c-1jammy) ... 2024-05-08T15:53:16.564 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package librdkafka1:amd64. 2024-05-08T15:53:16.590 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../017-librdkafka1_1.8.0-1build1_amd64.deb ... 2024-05-08T15:53:16.631 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2024-05-08T15:53:16.941 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package libreadline-dev:amd64. 2024-05-08T15:53:16.957 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../018-libreadline-dev_8.1.2-1_amd64.deb ... 2024-05-08T15:53:17.000 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking libreadline-dev:amd64 (8.1.2-1) ... 2024-05-08T15:53:17.303 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package liblua5.3-dev:amd64. 2024-05-08T15:53:17.329 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../019-liblua5.3-dev_5.3.6-1build1_amd64.deb ... 2024-05-08T15:53:17.369 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-05-08T15:53:17.673 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package lua5.1. 2024-05-08T15:53:17.699 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../020-lua5.1_5.1.5-8.1build4_amd64.deb ... 2024-05-08T15:53:17.739 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking lua5.1 (5.1.5-8.1build4) ... 2024-05-08T15:53:18.051 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package lua-any. 2024-05-08T15:53:18.066 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../021-lua-any_27ubuntu1_all.deb ... 2024-05-08T15:53:18.100 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking lua-any (27ubuntu1) ... 2024-05-08T15:53:18.328 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package zip. 2024-05-08T15:53:18.344 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../022-zip_3.0-12build2_amd64.deb ... 2024-05-08T15:53:18.386 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking zip (3.0-12build2) ... 2024-05-08T15:53:18.698 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package unzip. 2024-05-08T15:53:18.724 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../023-unzip_6.0-26ubuntu3.2_amd64.deb ... 2024-05-08T15:53:18.765 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking unzip (6.0-26ubuntu3.2) ... 2024-05-08T15:53:19.135 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package luarocks. 2024-05-08T15:53:19.162 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../024-luarocks_3.8.0+dfsg1-1_all.deb ... 2024-05-08T15:53:19.202 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking luarocks (3.8.0+dfsg1-1) ... 2024-05-08T15:53:19.509 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package librgw2. 2024-05-08T15:53:19.536 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../025-librgw2_19.0.0-2517-g95a4ad3c-1jammy_amd64.deb ... 2024-05-08T15:53:19.576 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking librgw2 (19.0.0-2517-g95a4ad3c-1jammy) ... 2024-05-08T15:53:20.312 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-rgw. 2024-05-08T15:53:20.327 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../026-python3-rgw_19.0.0-2517-g95a4ad3c-1jammy_amd64.deb ... 2024-05-08T15:53:20.362 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-rgw (19.0.0-2517-g95a4ad3c-1jammy) ... 2024-05-08T15:53:20.698 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package liboath0:amd64. 2024-05-08T15:53:20.725 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../027-liboath0_2.6.7-3build1_amd64.deb ... 2024-05-08T15:53:20.765 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking liboath0:amd64 (2.6.7-3build1) ... 2024-05-08T15:53:21.084 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package libradosstriper1. 2024-05-08T15:53:21.110 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../028-libradosstriper1_19.0.0-2517-g95a4ad3c-1jammy_amd64.deb ... 2024-05-08T15:53:21.151 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking libradosstriper1 (19.0.0-2517-g95a4ad3c-1jammy) ... 2024-05-08T15:53:21.579 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package ceph-common. 2024-05-08T15:53:21.606 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../029-ceph-common_19.0.0-2517-g95a4ad3c-1jammy_amd64.deb ... 2024-05-08T15:53:21.645 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking ceph-common (19.0.0-2517-g95a4ad3c-1jammy) ... 2024-05-08T15:53:22.961 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package ceph-base. 2024-05-08T15:53:22.987 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../030-ceph-base_19.0.0-2517-g95a4ad3c-1jammy_amd64.deb ... 2024-05-08T15:53:23.028 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking ceph-base (19.0.0-2517-g95a4ad3c-1jammy) ... 2024-05-08T15:53:23.565 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-jaraco.functools. 2024-05-08T15:53:23.592 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../031-python3-jaraco.functools_3.4.0-2_all.deb ... 2024-05-08T15:53:23.623 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2024-05-08T15:53:23.893 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-cheroot. 2024-05-08T15:53:23.919 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../032-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2024-05-08T15:53:23.960 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-05-08T15:53:24.296 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-jaraco.classes. 2024-05-08T15:53:24.322 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../033-python3-jaraco.classes_3.2.1-3_all.deb ... 2024-05-08T15:53:24.362 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2024-05-08T15:53:24.665 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-jaraco.text. 2024-05-08T15:53:24.692 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../034-python3-jaraco.text_3.6.0-2_all.deb ... 2024-05-08T15:53:24.732 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2024-05-08T15:53:24.993 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-jaraco.collections. 2024-05-08T15:53:25.020 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../035-python3-jaraco.collections_3.4.0-2_all.deb ... 2024-05-08T15:53:25.060 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2024-05-08T15:53:25.329 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-tempora. 2024-05-08T15:53:25.356 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../036-python3-tempora_4.1.2-1_all.deb ... 2024-05-08T15:53:25.396 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-tempora (4.1.2-1) ... 2024-05-08T15:53:25.699 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-portend. 2024-05-08T15:53:25.725 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../037-python3-portend_3.0.0-1_all.deb ... 2024-05-08T15:53:25.765 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-portend (3.0.0-1) ... 2024-05-08T15:53:26.068 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-zc.lockfile. 2024-05-08T15:53:26.095 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../038-python3-zc.lockfile_2.0-1_all.deb ... 2024-05-08T15:53:26.135 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2024-05-08T15:53:26.379 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-cherrypy3. 2024-05-08T15:53:26.406 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../039-python3-cherrypy3_18.6.1-4_all.deb ... 2024-05-08T15:53:26.437 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2024-05-08T15:53:26.741 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-natsort. 2024-05-08T15:53:26.768 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../040-python3-natsort_8.0.2-1_all.deb ... 2024-05-08T15:53:26.808 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-natsort (8.0.2-1) ... 2024-05-08T15:53:27.103 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-logutils. 2024-05-08T15:53:27.129 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../041-python3-logutils_0.3.3-8_all.deb ... 2024-05-08T15:53:27.169 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-logutils (0.3.3-8) ... 2024-05-08T15:53:27.472 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-markupsafe. 2024-05-08T15:53:27.498 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../042-python3-markupsafe_2.0.1-2build1_amd64.deb ... 2024-05-08T15:53:27.538 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-markupsafe (2.0.1-2build1) ... 2024-05-08T15:53:27.850 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-mako. 2024-05-08T15:53:27.876 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../043-python3-mako_1.1.3+ds1-2ubuntu0.1_all.deb ... 2024-05-08T15:53:27.916 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-05-08T15:53:28.211 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-simplegeneric. 2024-05-08T15:53:28.227 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../044-python3-simplegeneric_0.8.1-3_all.deb ... 2024-05-08T15:53:28.261 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2024-05-08T15:53:28.531 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-singledispatch. 2024-05-08T15:53:28.557 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../045-python3-singledispatch_3.4.0.3-3_all.deb ... 2024-05-08T15:53:28.597 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-singledispatch (3.4.0.3-3) ... 2024-05-08T15:53:28.884 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-webob. 2024-05-08T15:53:28.910 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../046-python3-webob_1%3a1.8.6-1.1_all.deb ... 2024-05-08T15:53:28.950 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-webob (1:1.8.6-1.1) ... 2024-05-08T15:53:29.294 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-waitress. 2024-05-08T15:53:29.321 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../047-python3-waitress_1.4.4-1.1ubuntu1_all.deb ... 2024-05-08T15:53:29.364 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-waitress (1.4.4-1.1ubuntu1) ... 2024-05-08T15:53:29.672 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-tempita. 2024-05-08T15:53:29.699 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../048-python3-tempita_0.5.2-6ubuntu1_all.deb ... 2024-05-08T15:53:29.739 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-tempita (0.5.2-6ubuntu1) ... 2024-05-08T15:53:30.026 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-paste. 2024-05-08T15:53:30.052 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../049-python3-paste_3.5.0+dfsg1-1_all.deb ... 2024-05-08T15:53:30.083 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-paste (3.5.0+dfsg1-1) ... 2024-05-08T15:53:30.445 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2024-05-08T15:53:30.472 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../050-python-pastedeploy-tpl_2.1.1-1_all.deb ... 2024-05-08T15:53:30.512 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python-pastedeploy-tpl (2.1.1-1) ... 2024-05-08T15:53:30.798 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-pastedeploy. 2024-05-08T15:53:30.824 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../051-python3-pastedeploy_2.1.1-1_all.deb ... 2024-05-08T15:53:30.865 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-pastedeploy (2.1.1-1) ... 2024-05-08T15:53:31.118 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-webtest. 2024-05-08T15:53:31.144 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../052-python3-webtest_2.0.35-1_all.deb ... 2024-05-08T15:53:31.209 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-webtest (2.0.35-1) ... 2024-05-08T15:53:31.513 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-pecan. 2024-05-08T15:53:31.538 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../053-python3-pecan_1.3.3-4ubuntu2_all.deb ... 2024-05-08T15:53:31.578 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-pecan (1.3.3-4ubuntu2) ... 2024-05-08T15:53:31.950 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-certifi. 2024-05-08T15:53:31.977 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../054-python3-certifi_2020.6.20-1_all.deb ... 2024-05-08T15:53:32.016 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-certifi (2020.6.20-1) ... 2024-05-08T15:53:32.378 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-idna. 2024-05-08T15:53:32.404 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../055-python3-idna_3.3-1_all.deb ... 2024-05-08T15:53:32.444 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-idna (3.3-1) ... 2024-05-08T15:53:32.764 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-urllib3. 2024-05-08T15:53:32.790 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../056-python3-urllib3_1.26.5-1~exp1ubuntu0.1_all.deb ... 2024-05-08T15:53:32.830 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-05-08T15:53:33.151 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-requests. 2024-05-08T15:53:33.177 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../057-python3-requests_2.25.1+dfsg-2ubuntu0.1_all.deb ... 2024-05-08T15:53:33.217 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-05-08T15:53:33.528 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-werkzeug. 2024-05-08T15:53:33.555 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../058-python3-werkzeug_2.0.2+dfsg1-1ubuntu0.22.04.1_all.deb ... 2024-05-08T15:53:33.595 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2024-05-08T15:53:33.889 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2024-05-08T15:53:33.904 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../059-ceph-mgr-modules-core_19.0.0-2517-g95a4ad3c-1jammy_all.deb ... 2024-05-08T15:53:33.948 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking ceph-mgr-modules-core (19.0.0-2517-g95a4ad3c-1jammy) ... 2024-05-08T15:53:34.286 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2024-05-08T15:53:34.312 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../060-libsqlite3-mod-ceph_19.0.0-2517-g95a4ad3c-1jammy_amd64.deb ... 2024-05-08T15:53:34.344 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking libsqlite3-mod-ceph (19.0.0-2517-g95a4ad3c-1jammy) ... 2024-05-08T15:53:34.664 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-bcrypt. 2024-05-08T15:53:34.690 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../061-python3-bcrypt_3.2.0-1build1_amd64.deb ... 2024-05-08T15:53:34.730 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-bcrypt (3.2.0-1build1) ... 2024-05-08T15:53:35.050 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-openssl. 2024-05-08T15:53:35.076 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../062-python3-openssl_21.0.0-1_all.deb ... 2024-05-08T15:53:35.116 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-openssl (21.0.0-1) ... 2024-05-08T15:53:35.420 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package ceph-mgr. 2024-05-08T15:53:35.446 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../063-ceph-mgr_19.0.0-2517-g95a4ad3c-1jammy_amd64.deb ... 2024-05-08T15:53:35.477 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking ceph-mgr (19.0.0-2517-g95a4ad3c-1jammy) ... 2024-05-08T15:53:36.639 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package ceph-mon. 2024-05-08T15:53:36.666 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../064-ceph-mon_19.0.0-2517-g95a4ad3c-1jammy_amd64.deb ... 2024-05-08T15:53:36.706 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking ceph-mon (19.0.0-2517-g95a4ad3c-1jammy) ... 2024-05-08T15:53:37.318 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package ceph-osd. 2024-05-08T15:53:37.345 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../065-ceph-osd_19.0.0-2517-g95a4ad3c-1jammy_amd64.deb ... 2024-05-08T15:53:37.384 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking ceph-osd (19.0.0-2517-g95a4ad3c-1jammy) ... 2024-05-08T15:53:38.454 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package ceph. 2024-05-08T15:53:38.469 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../066-ceph_19.0.0-2517-g95a4ad3c-1jammy_amd64.deb ... 2024-05-08T15:53:38.504 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking ceph (19.0.0-2517-g95a4ad3c-1jammy) ... 2024-05-08T15:53:38.815 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package ceph-fuse. 2024-05-08T15:53:38.842 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../067-ceph-fuse_19.0.0-2517-g95a4ad3c-1jammy_amd64.deb ... 2024-05-08T15:53:38.881 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking ceph-fuse (19.0.0-2517-g95a4ad3c-1jammy) ... 2024-05-08T15:53:39.268 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package ceph-mds. 2024-05-08T15:53:39.295 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../068-ceph-mds_19.0.0-2517-g95a4ad3c-1jammy_amd64.deb ... 2024-05-08T15:53:39.335 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking ceph-mds (19.0.0-2517-g95a4ad3c-1jammy) ... 2024-05-08T15:53:39.721 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package cephadm. 2024-05-08T15:53:39.748 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../069-cephadm_19.0.0-2517-g95a4ad3c-1jammy_amd64.deb ... 2024-05-08T15:53:39.787 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking cephadm (19.0.0-2517-g95a4ad3c-1jammy) ... 2024-05-08T15:53:40.066 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-asyncssh. 2024-05-08T15:53:40.093 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../070-python3-asyncssh_2.5.0-1_all.deb ... 2024-05-08T15:53:40.132 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-asyncssh (2.5.0-1) ... 2024-05-08T15:53:40.451 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python-babel-localedata. 2024-05-08T15:53:40.467 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../071-python-babel-localedata_2.8.0+dfsg.1-7_all.deb ... 2024-05-08T15:53:40.502 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-05-08T15:53:41.757 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-babel. 2024-05-08T15:53:41.784 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../072-python3-babel_2.8.0+dfsg.1-7_all.deb ... 2024-05-08T15:53:41.823 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-babel (2.8.0+dfsg.1-7) ... 2024-05-08T15:53:42.193 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-jinja2. 2024-05-08T15:53:42.220 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../073-python3-jinja2_3.0.3-1ubuntu0.1_all.deb ... 2024-05-08T15:53:42.259 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-jinja2 (3.0.3-1ubuntu0.1) ... 2024-05-08T15:53:42.612 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2024-05-08T15:53:42.640 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../074-ceph-mgr-cephadm_19.0.0-2517-g95a4ad3c-1jammy_all.deb ... 2024-05-08T15:53:42.679 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking ceph-mgr-cephadm (19.0.0-2517-g95a4ad3c-1jammy) ... 2024-05-08T15:53:43.015 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-repoze.lru. 2024-05-08T15:53:43.043 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../075-python3-repoze.lru_0.7-2_all.deb ... 2024-05-08T15:53:43.082 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2024-05-08T15:53:43.393 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-routes. 2024-05-08T15:53:43.421 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../076-python3-routes_2.5.1-1ubuntu1_all.deb ... 2024-05-08T15:53:43.460 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2024-05-08T15:53:43.796 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2024-05-08T15:53:43.824 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../077-ceph-mgr-dashboard_19.0.0-2517-g95a4ad3c-1jammy_all.deb ... 2024-05-08T15:53:43.862 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking ceph-mgr-dashboard (19.0.0-2517-g95a4ad3c-1jammy) ... 2024-05-08T15:53:45.135 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2024-05-08T15:53:45.162 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../078-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2024-05-08T15:53:45.201 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-05-08T15:53:45.738 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-joblib. 2024-05-08T15:53:45.765 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../079-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2024-05-08T15:53:45.804 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2024-05-08T15:53:46.141 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-threadpoolctl. 2024-05-08T15:53:46.169 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../080-python3-threadpoolctl_3.1.0-1_all.deb ... 2024-05-08T15:53:46.207 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2024-05-08T15:53:46.477 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-sklearn. 2024-05-08T15:53:46.505 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../081-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2024-05-08T15:53:46.543 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2024-05-08T15:53:47.303 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2024-05-08T15:53:47.331 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../082-ceph-mgr-diskprediction-local_19.0.0-2517-g95a4ad3c-1jammy_all.deb ... 2024-05-08T15:53:47.471 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking ceph-mgr-diskprediction-local (19.0.0-2517-g95a4ad3c-1jammy) ... 2024-05-08T15:53:49.010 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-cachetools. 2024-05-08T15:53:49.037 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../083-python3-cachetools_5.0.0-1_all.deb ... 2024-05-08T15:53:49.098 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2024-05-08T15:53:49.379 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-pyasn1. 2024-05-08T15:53:49.406 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../084-python3-pyasn1_0.4.8-1_all.deb ... 2024-05-08T15:53:49.437 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-pyasn1 (0.4.8-1) ... 2024-05-08T15:53:49.707 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-pyasn1-modules. 2024-05-08T15:53:49.723 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../085-python3-pyasn1-modules_0.2.1-1_all.deb ... 2024-05-08T15:53:49.757 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-pyasn1-modules (0.2.1-1) ... 2024-05-08T15:53:50.043 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-rsa. 2024-05-08T15:53:50.071 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../086-python3-rsa_4.8-1_all.deb ... 2024-05-08T15:53:50.110 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-rsa (4.8-1) ... 2024-05-08T15:53:50.430 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-google-auth. 2024-05-08T15:53:50.457 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../087-python3-google-auth_1.5.1-3_all.deb ... 2024-05-08T15:53:50.496 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2024-05-08T15:53:50.817 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-requests-oauthlib. 2024-05-08T15:53:50.832 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../088-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2024-05-08T15:53:50.883 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-05-08T15:53:51.186 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-websocket. 2024-05-08T15:53:51.214 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../089-python3-websocket_1.2.3-1_all.deb ... 2024-05-08T15:53:51.253 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-websocket (1.2.3-1) ... 2024-05-08T15:53:51.556 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-kubernetes. 2024-05-08T15:53:51.583 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../090-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2024-05-08T15:53:51.614 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2024-05-08T15:53:52.312 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2024-05-08T15:53:52.328 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../091-ceph-mgr-k8sevents_19.0.0-2517-g95a4ad3c-1jammy_all.deb ... 2024-05-08T15:53:52.370 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking ceph-mgr-k8sevents (19.0.0-2517-g95a4ad3c-1jammy) ... 2024-05-08T15:53:52.864 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package libonig5:amd64. 2024-05-08T15:53:52.892 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../092-libonig5_6.9.7.1-2build1_amd64.deb ... 2024-05-08T15:53:52.931 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2024-05-08T15:53:53.260 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package libjq1:amd64. 2024-05-08T15:53:53.287 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../093-libjq1_1.6-2.1ubuntu3_amd64.deb ... 2024-05-08T15:53:53.326 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-05-08T15:53:53.587 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package jq. 2024-05-08T15:53:53.614 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../094-jq_1.6-2.1ubuntu3_amd64.deb ... 2024-05-08T15:53:53.653 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking jq (1.6-2.1ubuntu3) ... 2024-05-08T15:53:53.897 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package socat. 2024-05-08T15:53:53.913 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../095-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2024-05-08T15:53:53.947 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2024-05-08T15:53:54.234 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package xmlstarlet. 2024-05-08T15:53:54.250 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../096-xmlstarlet_1.6.1-2.1_amd64.deb ... 2024-05-08T15:53:54.284 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2024-05-08T15:53:54.647 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package ceph-test. 2024-05-08T15:53:54.674 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../097-ceph-test_19.0.0-2517-g95a4ad3c-1jammy_amd64.deb ... 2024-05-08T15:53:54.713 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking ceph-test (19.0.0-2517-g95a4ad3c-1jammy) ... 2024-05-08T15:53:57.051 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package ceph-volume. 2024-05-08T15:53:57.078 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../098-ceph-volume_19.0.0-2517-g95a4ad3c-1jammy_all.deb ... 2024-05-08T15:53:57.117 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking ceph-volume (19.0.0-2517-g95a4ad3c-1jammy) ... 2024-05-08T15:53:57.438 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package libcephfs-dev. 2024-05-08T15:53:57.454 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../099-libcephfs-dev_19.0.0-2517-g95a4ad3c-1jammy_amd64.deb ... 2024-05-08T15:53:57.496 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking libcephfs-dev (19.0.0-2517-g95a4ad3c-1jammy) ... 2024-05-08T15:53:57.891 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package lua-socket:amd64. 2024-05-08T15:53:57.918 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../100-lua-socket_3.0~rc1+git+ac3201d-6_amd64.deb ... 2024-05-08T15:53:57.957 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-05-08T15:53:58.312 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package lua-sec:amd64. 2024-05-08T15:53:58.339 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../101-lua-sec_1.0.2-1_amd64.deb ... 2024-05-08T15:53:58.378 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking lua-sec:amd64 (1.0.2-1) ... 2024-05-08T15:53:58.699 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package nvme-cli. 2024-05-08T15:53:58.726 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../102-nvme-cli_1.16-3ubuntu0.1_amd64.deb ... 2024-05-08T15:53:58.757 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking nvme-cli (1.16-3ubuntu0.1) ... 2024-05-08T15:53:59.179 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package pkg-config. 2024-05-08T15:53:59.207 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../103-pkg-config_0.29.2-1ubuntu3_amd64.deb ... 2024-05-08T15:53:59.245 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking pkg-config (0.29.2-1ubuntu3) ... 2024-05-08T15:53:59.507 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python-asyncssh-doc. 2024-05-08T15:53:59.534 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../104-python-asyncssh-doc_2.5.0-1_all.deb ... 2024-05-08T15:53:59.573 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python-asyncssh-doc (2.5.0-1) ... 2024-05-08T15:54:00.076 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-iniconfig. 2024-05-08T15:54:00.092 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../105-python3-iniconfig_1.1.1-2_all.deb ... 2024-05-08T15:54:00.134 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2024-05-08T15:54:00.429 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-pastescript. 2024-05-08T15:54:00.457 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../106-python3-pastescript_2.0.2-4_all.deb ... 2024-05-08T15:54:00.495 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2024-05-08T15:54:00.817 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-pluggy. 2024-05-08T15:54:00.831 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../107-python3-pluggy_0.13.0-7.1_all.deb ... 2024-05-08T15:54:00.873 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2024-05-08T15:54:01.144 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-psutil. 2024-05-08T15:54:01.171 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../108-python3-psutil_5.9.0-1build1_amd64.deb ... 2024-05-08T15:54:01.210 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2024-05-08T15:54:01.555 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-py. 2024-05-08T15:54:01.582 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../109-python3-py_1.10.0-1_all.deb ... 2024-05-08T15:54:01.629 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-py (1.10.0-1) ... 2024-05-08T15:54:01.949 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-pygments. 2024-05-08T15:54:01.977 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../110-python3-pygments_2.11.2+dfsg-2_all.deb ... 2024-05-08T15:54:02.016 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-pygments (2.11.2+dfsg-2) ... 2024-05-08T15:54:02.497 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-pyinotify. 2024-05-08T15:54:02.525 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../111-python3-pyinotify_0.9.6-1.3_all.deb ... 2024-05-08T15:54:02.563 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-pyinotify (0.9.6-1.3) ... 2024-05-08T15:54:02.841 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-toml. 2024-05-08T15:54:02.869 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../112-python3-toml_0.10.2-1_all.deb ... 2024-05-08T15:54:02.907 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-toml (0.10.2-1) ... 2024-05-08T15:54:03.194 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-pytest. 2024-05-08T15:54:03.221 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../113-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2024-05-08T15:54:03.260 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2024-05-08T15:54:03.589 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-simplejson. 2024-05-08T15:54:03.616 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../114-python3-simplejson_3.17.6-1build1_amd64.deb ... 2024-05-08T15:54:03.655 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2024-05-08T15:54:03.900 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package qttranslations5-l10n. 2024-05-08T15:54:03.927 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../115-qttranslations5-l10n_5.15.3-1_all.deb ... 2024-05-08T15:54:03.966 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2024-05-08T15:54:04.695 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package radosgw. 2024-05-08T15:54:04.723 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../116-radosgw_19.0.0-2517-g95a4ad3c-1jammy_amd64.deb ... 2024-05-08T15:54:04.761 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking radosgw (19.0.0-2517-g95a4ad3c-1jammy) ... 2024-05-08T15:54:05.581 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package rbd-fuse. 2024-05-08T15:54:05.597 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../117-rbd-fuse_19.0.0-2517-g95a4ad3c-1jammy_amd64.deb ... 2024-05-08T15:54:05.639 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking rbd-fuse (19.0.0-2517-g95a4ad3c-1jammy) ... 2024-05-08T15:54:05.935 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2024-05-08T15:54:06.177 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2024-05-08T15:54:06.303 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up nvme-cli (1.16-3ubuntu0.1) ... 2024-05-08T15:54:06.568 INFO:teuthology.orchestra.run.smithi172.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service -> /lib/systemd/system/nvmefc-boot-connections.service. 2024-05-08T15:54:06.915 INFO:teuthology.orchestra.run.smithi172.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service -> /lib/systemd/system/nvmf-autoconnect.service. 2024-05-08T15:54:07.329 INFO:teuthology.orchestra.run.smithi172.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2024-05-08T15:54:07.397 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up cephadm (19.0.0-2517-g95a4ad3c-1jammy) ... 2024-05-08T15:54:07.930 INFO:teuthology.orchestra.run.smithi172.stdout:Adding system user cephadm....done 2024-05-08T15:54:08.087 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-waitress (1.4.4-1.1ubuntu1) ... 2024-05-08T15:54:08.363 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2024-05-08T15:54:08.622 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python-asyncssh-doc (2.5.0-1) ... 2024-05-08T15:54:08.740 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2024-05-08T15:54:08.983 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-repoze.lru (0.7-2) ... 2024-05-08T15:54:09.234 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up liboath0:amd64 (2.6.7-3build1) ... 2024-05-08T15:54:09.360 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-py (1.10.0-1) ... 2024-05-08T15:54:09.670 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2024-05-08T15:54:10.047 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-cachetools (5.0.0-1) ... 2024-05-08T15:54:10.298 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-openssl (21.0.0-1) ... 2024-05-08T15:54:10.596 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up unzip (6.0-26ubuntu3.2) ... 2024-05-08T15:54:10.834 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-bcrypt (3.2.0-1build1) ... 2024-05-08T15:54:11.077 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-pyinotify (0.9.6-1.3) ... 2024-05-08T15:54:11.345 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2024-05-08T15:54:11.596 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-ceph-argparse (19.0.0-2517-g95a4ad3c-1jammy) ... 2024-05-08T15:54:12.014 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-05-08T15:54:12.139 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-05-08T15:54:12.240 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up libreadline-dev:amd64 (8.1.2-1) ... 2024-05-08T15:54:12.341 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-markupsafe (2.0.1-2build1) ... 2024-05-08T15:54:12.934 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up lua5.1 (5.1.5-8.1build4) ... 2024-05-08T15:54:13.320 INFO:teuthology.orchestra.run.smithi172.stdout:update-alternatives: using /usr/bin/lua5.1 to provide /usr/bin/lua (lua-interpreter) in auto mode 2024-05-08T15:54:13.349 INFO:teuthology.orchestra.run.smithi172.stdout:update-alternatives: using /usr/bin/luac5.1 to provide /usr/bin/luac (lua-compiler) in auto mode 2024-05-08T15:54:13.412 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-05-08T15:54:13.538 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2024-05-08T15:54:13.906 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-05-08T15:54:14.007 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-natsort (8.0.2-1) ... 2024-05-08T15:54:14.267 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2024-05-08T15:54:14.526 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2024-05-08T15:54:14.836 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up zip (3.0-12build2) ... 2024-05-08T15:54:14.961 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-pygments (2.11.2+dfsg-2) ... 2024-05-08T15:54:15.663 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-tempita (0.5.2-6ubuntu1) ... 2024-05-08T15:54:15.931 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python-pastedeploy-tpl (2.1.1-1) ... 2024-05-08T15:54:16.032 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2024-05-08T15:54:16.133 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2024-05-08T15:54:16.409 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-asyncssh (2.5.0-1) ... 2024-05-08T15:54:16.794 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-certifi (2020.6.20-1) ... 2024-05-08T15:54:17.045 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-paste (3.5.0+dfsg1-1) ... 2024-05-08T15:54:17.421 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-05-08T15:54:17.714 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2024-05-08T15:54:18.057 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2024-05-08T15:54:18.308 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2024-05-08T15:54:18.434 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-ceph-common (19.0.0-2517-g95a4ad3c-1jammy) ... 2024-05-08T15:54:18.727 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-idna (3.3-1) ... 2024-05-08T15:54:19.054 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2024-05-08T15:54:20.234 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up pkg-config (0.29.2-1ubuntu3) ... 2024-05-08T15:54:20.469 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-05-08T15:54:20.603 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-toml (0.10.2-1) ... 2024-05-08T15:54:20.871 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2024-05-08T15:54:20.997 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2024-05-08T15:54:21.106 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-05-08T15:54:21.399 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2024-05-08T15:54:21.650 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2024-05-08T15:54:21.918 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-05-08T15:54:22.044 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-pyasn1 (0.4.8-1) ... 2024-05-08T15:54:22.346 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-singledispatch (3.4.0.3-3) ... 2024-05-08T15:54:22.613 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-logutils (0.3.3-8) ... 2024-05-08T15:54:22.873 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-tempora (4.1.2-1) ... 2024-05-08T15:54:23.146 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2024-05-08T15:54:23.375 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-prettytable (2.5.0-2) ... 2024-05-08T15:54:23.643 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-05-08T15:54:23.777 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-websocket (1.2.3-1) ... 2024-05-08T15:54:24.070 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2024-05-08T15:54:24.196 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-05-08T15:54:24.489 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-webob (1:1.8.6-1.1) ... 2024-05-08T15:54:24.781 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2024-05-08T15:54:25.016 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-05-08T15:54:25.125 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up lua-sec:amd64 (1.0.2-1) ... 2024-05-08T15:54:25.234 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-babel (2.8.0+dfsg.1-7) ... 2024-05-08T15:54:25.316 INFO:teuthology.orchestra.run.smithi172.stdout:update-alternatives: using /usr/bin/pybabel-python3 to provide /usr/bin/pybabel (pybabel) in auto mode 2024-05-08T15:54:25.561 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-05-08T15:54:25.687 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2024-05-08T15:54:26.072 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-pastedeploy (2.1.1-1) ... 2024-05-08T15:54:26.323 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up lua-any (27ubuntu1) ... 2024-05-08T15:54:26.424 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-portend (3.0.0-1) ... 2024-05-08T15:54:26.675 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-05-08T15:54:26.801 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-jinja2 (3.0.3-1ubuntu0.1) ... 2024-05-08T15:54:27.144 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-pyasn1-modules (0.2.1-1) ... 2024-05-08T15:54:27.453 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-05-08T15:54:27.729 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up jq (1.6-2.1ubuntu3) ... 2024-05-08T15:54:27.855 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-rsa (4.8-1) ... 2024-05-08T15:54:28.123 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-webtest (2.0.35-1) ... 2024-05-08T15:54:28.383 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-05-08T15:54:28.711 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2024-05-08T15:54:29.104 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-pastescript (2.0.2-4) ... 2024-05-08T15:54:29.389 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-pecan (1.3.3-4ubuntu2) ... 2024-05-08T15:54:29.725 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-05-08T15:54:29.826 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up librados2 (19.0.0-2517-g95a4ad3c-1jammy) ... 2024-05-08T15:54:29.952 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up libsqlite3-mod-ceph (19.0.0-2517-g95a4ad3c-1jammy) ... 2024-05-08T15:54:30.078 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up luarocks (3.8.0+dfsg1-1) ... 2024-05-08T15:54:30.414 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up libcephfs2 (19.0.0-2517-g95a4ad3c-1jammy) ... 2024-05-08T15:54:30.540 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up libradosstriper1 (19.0.0-2517-g95a4ad3c-1jammy) ... 2024-05-08T15:54:30.658 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-google-auth (1.5.1-3) ... 2024-05-08T15:54:30.926 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up librbd1 (19.0.0-2517-g95a4ad3c-1jammy) ... 2024-05-08T15:54:31.044 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up ceph-mgr-modules-core (19.0.0-2517-g95a4ad3c-1jammy) ... 2024-05-08T15:54:31.170 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up ceph-fuse (19.0.0-2517-g95a4ad3c-1jammy) ... 2024-05-08T15:54:31.380 INFO:teuthology.orchestra.run.smithi172.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-05-08T15:54:31.381 INFO:teuthology.orchestra.run.smithi172.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-05-08T15:54:31.871 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up libcephfs-dev (19.0.0-2517-g95a4ad3c-1jammy) ... 2024-05-08T15:54:31.997 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-rados (19.0.0-2517-g95a4ad3c-1jammy) ... 2024-05-08T15:54:32.148 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up librgw2 (19.0.0-2517-g95a4ad3c-1jammy) ... 2024-05-08T15:54:32.274 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2024-05-08T15:54:33.411 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-rbd (19.0.0-2517-g95a4ad3c-1jammy) ... 2024-05-08T15:54:33.537 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up rbd-fuse (19.0.0-2517-g95a4ad3c-1jammy) ... 2024-05-08T15:54:33.663 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-rgw (19.0.0-2517-g95a4ad3c-1jammy) ... 2024-05-08T15:54:33.764 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-cephfs (19.0.0-2517-g95a4ad3c-1jammy) ... 2024-05-08T15:54:33.865 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up ceph-common (19.0.0-2517-g95a4ad3c-1jammy) ... 2024-05-08T15:54:34.416 INFO:teuthology.orchestra.run.smithi172.stdout:Adding group ceph....done 2024-05-08T15:54:34.726 INFO:teuthology.orchestra.run.smithi172.stdout:Adding system user ceph....done 2024-05-08T15:54:34.895 INFO:teuthology.orchestra.run.smithi172.stdout:Setting system user ceph properties....done 2024-05-08T15:54:34.910 INFO:teuthology.orchestra.run.smithi172.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2024-05-08T15:54:35.056 INFO:teuthology.orchestra.run.smithi172.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2024-05-08T15:54:35.350 INFO:teuthology.orchestra.run.smithi172.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2024-05-08T15:54:35.783 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up ceph-test (19.0.0-2517-g95a4ad3c-1jammy) ... 2024-05-08T15:54:35.884 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up radosgw (19.0.0-2517-g95a4ad3c-1jammy) ... 2024-05-08T15:54:36.312 INFO:teuthology.orchestra.run.smithi172.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-05-08T15:54:36.312 INFO:teuthology.orchestra.run.smithi172.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-05-08T15:54:36.736 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up ceph-base (19.0.0-2517-g95a4ad3c-1jammy) ... 2024-05-08T15:54:37.079 INFO:teuthology.orchestra.run.smithi172.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2024-05-08T15:54:37.537 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up ceph-mds (19.0.0-2517-g95a4ad3c-1jammy) ... 2024-05-08T15:54:37.761 INFO:teuthology.orchestra.run.smithi172.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-05-08T15:54:37.761 INFO:teuthology.orchestra.run.smithi172.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-05-08T15:54:38.206 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up ceph-mgr (19.0.0-2517-g95a4ad3c-1jammy) ... 2024-05-08T15:54:38.408 INFO:teuthology.orchestra.run.smithi172.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-05-08T15:54:38.409 INFO:teuthology.orchestra.run.smithi172.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-05-08T15:54:38.866 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up ceph-osd (19.0.0-2517-g95a4ad3c-1jammy) ... 2024-05-08T15:54:39.127 INFO:teuthology.orchestra.run.smithi172.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-05-08T15:54:39.127 INFO:teuthology.orchestra.run.smithi172.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-05-08T15:54:39.559 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up ceph-mgr-k8sevents (19.0.0-2517-g95a4ad3c-1jammy) ... 2024-05-08T15:54:39.668 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up ceph-mgr-diskprediction-local (19.0.0-2517-g95a4ad3c-1jammy) ... 2024-05-08T15:54:39.811 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up ceph-mon (19.0.0-2517-g95a4ad3c-1jammy) ... 2024-05-08T15:54:40.023 INFO:teuthology.orchestra.run.smithi172.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-05-08T15:54:40.023 INFO:teuthology.orchestra.run.smithi172.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-05-08T15:54:40.471 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up ceph-mgr-cephadm (19.0.0-2517-g95a4ad3c-1jammy) ... 2024-05-08T15:54:40.597 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up ceph (19.0.0-2517-g95a4ad3c-1jammy) ... 2024-05-08T15:54:40.698 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up ceph-mgr-dashboard (19.0.0-2517-g95a4ad3c-1jammy) ... 2024-05-08T15:54:40.824 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up ceph-volume (19.0.0-2517-g95a4ad3c-1jammy) ... 2024-05-08T15:54:41.160 INFO:teuthology.orchestra.run.smithi172.stdout:Processing triggers for mailcap (3.70+nmu1ubuntu1) ... 2024-05-08T15:54:41.269 INFO:teuthology.orchestra.run.smithi172.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2024-05-08T15:54:41.405 INFO:teuthology.orchestra.run.smithi172.stdout:Processing triggers for man-db (2.10.2-1) ... 2024-05-08T15:54:42.589 INFO:teuthology.orchestra.run.smithi172.stdout:Processing triggers for install-info (6.8-4build1) ... 2024-05-08T15:54:43.259 INFO:teuthology.orchestra.run.smithi172.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-05-08T15:54:43.265 DEBUG:teuthology.parallel:result is None 2024-05-08T15:54:43.265 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=95a4ad3c90f2f12f920e87725ad8d9002f948722 2024-05-08T15:54:43.405 DEBUG:teuthology.orchestra.run.smithi172:> dpkg-query -W -f '${Version}' ceph 2024-05-08T15:54:43.427 INFO:teuthology.orchestra.run.smithi172.stdout:19.0.0-2517-g95a4ad3c-1jammy 2024-05-08T15:54:43.428 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2517-g95a4ad3c-1jammy 2024-05-08T15:54:43.428 INFO:teuthology.task.install:The correct ceph version 19.0.0-2517-g95a4ad3c-1jammy is installed. 2024-05-08T15:54:43.431 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-05-08T15:54:43.431 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2024-05-08T15:54:43.431 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-05-08T15:54:43.484 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-05-08T15:54:43.484 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2024-05-08T15:54:43.484 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd of=/usr/bin/daemon-helper 2024-05-08T15:54:43.539 DEBUG:teuthology.orchestra.run.smithi172:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-05-08T15:54:43.595 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-05-08T15:54:43.595 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2024-05-08T15:54:43.595 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd of=/usr/bin/adjust-ulimits 2024-05-08T15:54:43.656 DEBUG:teuthology.orchestra.run.smithi172:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-05-08T15:54:43.711 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-05-08T15:54:43.711 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2024-05-08T15:54:43.711 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd of=/usr/bin/stdin-killer 2024-05-08T15:54:43.767 DEBUG:teuthology.orchestra.run.smithi172:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-05-08T15:54:43.822 INFO:teuthology.run_tasks:Running task workunit... 2024-05-08T15:54:43.839 INFO:tasks.workunit:Pulling workunits from ref 95a4ad3c90f2f12f920e87725ad8d9002f948722 2024-05-08T15:54:43.840 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-05-08T15:54:43.840 INFO:tasks.workunit:timeout=3h 2024-05-08T15:54:43.840 INFO:tasks.workunit:cleanup=True 2024-05-08T15:54:43.840 DEBUG:teuthology.orchestra.run.smithi172:> stat -- /home/ubuntu/cephtest/mnt.0 2024-05-08T15:54:43.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-08T15:54:43.867 INFO:teuthology.orchestra.run.smithi172.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-05-08T15:54:43.868 DEBUG:teuthology.orchestra.run.smithi172:> mkdir -- /home/ubuntu/cephtest/mnt.0 2024-05-08T15:54:43.915 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2024-05-08T15:54:43.916 DEBUG:teuthology.orchestra.run.smithi172:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2024-05-08T15:54:43.963 DEBUG:teuthology.orchestra.run.smithi172:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://git.ceph.com/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 95a4ad3c90f2f12f920e87725ad8d9002f948722 2024-05-08T15:54:44.013 INFO:tasks.workunit.client.0.smithi172.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-05-08T15:56:06.273 INFO:tasks.workunit.client.0.smithi172.stderr:Note: switching to '95a4ad3c90f2f12f920e87725ad8d9002f948722'. 2024-05-08T15:56:06.273 INFO:tasks.workunit.client.0.smithi172.stderr: 2024-05-08T15:56:06.273 INFO:tasks.workunit.client.0.smithi172.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-05-08T15:56:06.273 INFO:tasks.workunit.client.0.smithi172.stderr:changes and commit them, and you can discard any commits you make in this 2024-05-08T15:56:06.273 INFO:tasks.workunit.client.0.smithi172.stderr:state without impacting any branches by switching back to a branch. 2024-05-08T15:56:06.273 INFO:tasks.workunit.client.0.smithi172.stderr: 2024-05-08T15:56:06.273 INFO:tasks.workunit.client.0.smithi172.stderr:If you want to create a new branch to retain commits you create, you may 2024-05-08T15:56:06.273 INFO:tasks.workunit.client.0.smithi172.stderr:do so (now or later) by using -c with the switch command. Example: 2024-05-08T15:56:06.274 INFO:tasks.workunit.client.0.smithi172.stderr: 2024-05-08T15:56:06.274 INFO:tasks.workunit.client.0.smithi172.stderr: git switch -c 2024-05-08T15:56:06.274 INFO:tasks.workunit.client.0.smithi172.stderr: 2024-05-08T15:56:06.274 INFO:tasks.workunit.client.0.smithi172.stderr:Or undo this operation with: 2024-05-08T15:56:06.274 INFO:tasks.workunit.client.0.smithi172.stderr: 2024-05-08T15:56:06.274 INFO:tasks.workunit.client.0.smithi172.stderr: git switch - 2024-05-08T15:56:06.274 INFO:tasks.workunit.client.0.smithi172.stderr: 2024-05-08T15:56:06.274 INFO:tasks.workunit.client.0.smithi172.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-05-08T15:56:06.274 INFO:tasks.workunit.client.0.smithi172.stderr: 2024-05-08T15:56:06.274 INFO:tasks.workunit.client.0.smithi172.stderr:HEAD is now at 95a4ad3c90f Merge branch 'wip-rf-reinstate-q-squid' of https://github.com/ronen-fr/ceph into wip-lflores-testing-2-2024-05-07-1606-squid 2024-05-08T15:56:06.279 DEBUG:teuthology.orchestra.run.smithi172:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/standalone && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-05-08T15:56:06.328 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2024-05-08T15:56:06.328 DEBUG:teuthology.orchestra.run.smithi172:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-05-08T15:56:06.375 INFO:tasks.workunit:Running workunits matching erasure-code on client.0... 2024-05-08T15:56:06.375 INFO:tasks.workunit:Running workunit erasure-code/test-erasure-code-plugins.sh... 2024-05-08T15:56:06.375 DEBUG:teuthology.orchestra.run.smithi172:workunit test erasure-code/test-erasure-code-plugins.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=95a4ad3c90f2f12f920e87725ad8d9002f948722 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh 2024-05-08T15:56:06.427 INFO:tasks.workunit.client.0.smithi172.stderr:+ source /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh 2024-05-08T15:56:06.428 INFO:tasks.workunit.client.0.smithi172.stderr:++ TIMEOUT=300 2024-05-08T15:56:06.428 INFO:tasks.workunit.client.0.smithi172.stderr:++ WAIT_FOR_CLEAN_TIMEOUT=90 2024-05-08T15:56:06.428 INFO:tasks.workunit.client.0.smithi172.stderr:++ MAX_TIMEOUT=15 2024-05-08T15:56:06.428 INFO:tasks.workunit.client.0.smithi172.stderr:++ PG_NUM=4 2024-05-08T15:56:06.428 INFO:tasks.workunit.client.0.smithi172.stderr:++ TMPDIR=/tmp 2024-05-08T15:56:06.428 INFO:tasks.workunit.client.0.smithi172.stderr:++ CEPH_BUILD_VIRTUALENV=/tmp 2024-05-08T15:56:06.428 INFO:tasks.workunit.client.0.smithi172.stderr:++ TESTDIR=/home/ubuntu/cephtest 2024-05-08T15:56:06.428 INFO:tasks.workunit.client.0.smithi172.stderr:++ type xmlstarlet 2024-05-08T15:56:06.428 INFO:tasks.workunit.client.0.smithi172.stderr:++ XMLSTARLET=xmlstarlet 2024-05-08T15:56:06.428 INFO:tasks.workunit.client.0.smithi172.stderr:+++ uname 2024-05-08T15:56:06.429 INFO:tasks.workunit.client.0.smithi172.stderr:++ '[' Linux = FreeBSD ']' 2024-05-08T15:56:06.429 INFO:tasks.workunit.client.0.smithi172.stderr:++ SED=sed 2024-05-08T15:56:06.429 INFO:tasks.workunit.client.0.smithi172.stderr:++ AWK=awk 2024-05-08T15:56:06.429 INFO:tasks.workunit.client.0.smithi172.stderr:+++ stty -a 2024-05-08T15:56:06.429 INFO:tasks.workunit.client.0.smithi172.stderr:+++ head -1 2024-05-08T15:56:06.429 INFO:tasks.workunit.client.0.smithi172.stderr:+++ sed -e 's/.*columns \([0-9]*\).*/\1/' 2024-05-08T15:56:06.439 INFO:tasks.workunit.client.0.smithi172.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-05-08T15:56:06.440 INFO:tasks.workunit.client.0.smithi172.stderr:++ termwidth= 2024-05-08T15:56:06.440 INFO:tasks.workunit.client.0.smithi172.stderr:++ '[' -n '' -a '' '!=' 0 ']' 2024-05-08T15:56:06.440 INFO:tasks.workunit.client.0.smithi172.stderr:++ DIFFCOLOPTS='-y ' 2024-05-08T15:56:06.440 INFO:tasks.workunit.client.0.smithi172.stderr:++ KERNCORE=kernel.core_pattern 2024-05-08T15:56:06.440 INFO:tasks.workunit.client.0.smithi172.stderr:++ EXTRA_OPTS= 2024-05-08T15:56:06.443 INFO:tasks.workunit.client.0.smithi172.stderr:++ test '' = TESTS 2024-05-08T15:56:06.443 INFO:tasks.workunit.client.0.smithi172.stderr:++ uname -m 2024-05-08T15:56:06.446 INFO:tasks.workunit.client.0.smithi172.stderr:+ arch=x86_64 2024-05-08T15:56:06.446 INFO:tasks.workunit.client.0.smithi172.stderr:+ case $arch in 2024-05-08T15:56:06.446 INFO:tasks.workunit.client.0.smithi172.stderr:+ legacy_jerasure_plugins=(jerasure_generic jerasure_sse3 jerasure_sse4) 2024-05-08T15:56:06.446 INFO:tasks.workunit.client.0.smithi172.stderr:+ legacy_shec_plugins=(shec_generic shec_sse3 shec_sse4) 2024-05-08T15:56:06.446 INFO:tasks.workunit.client.0.smithi172.stderr:+ plugins=(jerasure shec lrc isa) 2024-05-08T15:56:06.446 INFO:tasks.workunit.client.0.smithi172.stderr:+ main test-erasure-code-plugins 2024-05-08T15:56:06.446 INFO:tasks.workunit.client.0.smithi172.stderr:+ local dir=td/test-erasure-code-plugins 2024-05-08T15:56:06.446 INFO:tasks.workunit.client.0.smithi172.stderr:+ shift 2024-05-08T15:56:06.446 INFO:tasks.workunit.client.0.smithi172.stderr:+ shopt -s -o xtrace 2024-05-08T15:56:06.446 INFO:tasks.workunit.client.0.smithi172.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-05-08T15:56:06.447 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2307: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-05-08T15:56:06.447 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2307: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-05-08T15:56:06.447 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2308: main: export PYTHONWARNINGS=ignore 2024-05-08T15:56:06.447 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2308: main: PYTHONWARNINGS=ignore 2024-05-08T15:56:06.447 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2309: main: export CEPH_CONF=/dev/null 2024-05-08T15:56:06.447 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2309: main: CEPH_CONF=/dev/null 2024-05-08T15:56:06.447 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2310: main: unset CEPH_ARGS 2024-05-08T15:56:06.447 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2312: main: local code 2024-05-08T15:56:06.447 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2313: main: run td/test-erasure-code-plugins 2024-05-08T15:56:06.447 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:26: run: local dir=td/test-erasure-code-plugins 2024-05-08T15:56:06.447 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:27: run: shift 2024-05-08T15:56:06.447 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:29: run: export CEPH_MON=127.0.0.1:17110 2024-05-08T15:56:06.447 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:29: run: CEPH_MON=127.0.0.1:17110 2024-05-08T15:56:06.447 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:30: run: export CEPH_ARGS 2024-05-08T15:56:06.447 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:31: run: uuidgen 2024-05-08T15:56:06.448 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:31: run: CEPH_ARGS+='--fsid=2a3a20f9-46fd-4090-ba60-6d102331c696 --auth-supported=none ' 2024-05-08T15:56:06.448 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:32: run: CEPH_ARGS+='--mon-host=127.0.0.1:17110 ' 2024-05-08T15:56:06.449 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:34: run: set 2024-05-08T15:56:06.449 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:34: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-05-08T15:56:06.450 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:34: run: local 'funcs=TEST_ec_profile_warning 2024-05-08T15:56:06.450 INFO:tasks.workunit.client.0.smithi172.stderr:TEST_preload_no_warning 2024-05-08T15:56:06.450 INFO:tasks.workunit.client.0.smithi172.stderr:TEST_preload_no_warning_default 2024-05-08T15:56:06.450 INFO:tasks.workunit.client.0.smithi172.stderr:TEST_preload_warning' 2024-05-08T15:56:06.450 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:35: run: for func in $funcs 2024-05-08T15:56:06.450 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:36: run: TEST_ec_profile_warning td/test-erasure-code-plugins 2024-05-08T15:56:06.450 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:92: TEST_ec_profile_warning: local dir=td/test-erasure-code-plugins 2024-05-08T15:56:06.450 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:94: TEST_ec_profile_warning: setup td/test-erasure-code-plugins 2024-05-08T15:56:06.450 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/test-erasure-code-plugins 2024-05-08T15:56:06.450 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/test-erasure-code-plugins 2024-05-08T15:56:06.450 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-code-plugins 2024-05-08T15:56:06.450 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-05-08T15:56:06.450 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-code-plugins KILL 2024-05-08T15:56:06.450 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-08T15:56:06.451 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-08T15:56:06.451 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-08T15:56:06.451 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-08T15:56:06.451 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-08T15:56:06.451 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-08T15:56:06.451 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-05-08T15:56:06.452 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-05-08T15:56:06.452 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-05-08T15:56:06.452 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-05-08T15:56:06.452 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-05-08T15:56:06.453 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-05-08T15:56:06.453 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-08T15:56:06.453 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-05-08T15:56:06.454 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-05-08T15:56:06.454 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-08T15:56:06.454 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-05-08T15:56:06.456 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-05-08T15:56:06.457 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-code-plugins 2024-05-08T15:56:06.457 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-05-08T15:56:06.457 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T15:56:06.457 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20299 2024-05-08T15:56:06.457 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.20299 2024-05-08T15:56:06.458 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-05-08T15:56:06.458 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-05-08T15:56:06.458 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/test-erasure-code-plugins 2024-05-08T15:56:06.458 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-05-08T15:56:06.458 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T15:56:06.458 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20299 2024-05-08T15:56:06.458 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.20299 2024-05-08T15:56:06.460 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-05-08T15:56:06.460 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2024-05-08T15:56:06.460 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2024-05-08T15:56:06.460 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-05-08T15:56:06.460 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/test-erasure-code-plugins 1' TERM HUP INT 2024-05-08T15:56:06.460 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:95: TEST_ec_profile_warning: run_mon td/test-erasure-code-plugins a 2024-05-08T15:56:06.461 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/test-erasure-code-plugins 2024-05-08T15:56:06.461 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-05-08T15:56:06.461 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-05-08T15:56:06.461 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-05-08T15:56:06.461 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/test-erasure-code-plugins/a 2024-05-08T15:56:06.461 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-code-plugins/a --run-dir=td/test-erasure-code-plugins 2024-05-08T15:56:06.966 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-05-08T15:56:06.966 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-08T15:56:06.966 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-08T15:56:06.966 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-08T15:56:06.967 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T15:56:06.967 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20299 2024-05-08T15:56:06.970 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.20299/$cluster-$name.asok' 2024-05-08T15:56:06.970 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-code-plugins/a '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.20299/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-code-plugins/log --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-05-08T15:56:07.176 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-05-08T15:56:07.177 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-05-08T15:56:07.177 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-05-08T15:56:07.177 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-05-08T15:56:07.177 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-05-08T15:56:07.179 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-05-08T15:56:07.179 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-05-08T15:56:07.179 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-05-08T15:56:07.179 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-05-08T15:56:07.179 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-08T15:56:07.179 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T15:56:07.179 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20299 2024-05-08T15:56:07.179 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20299/ceph-mon.a.asok 2024-05-08T15:56:07.180 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-05-08T15:56:07.180 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.20299/ceph-mon.a.asok config get fsid 2024-05-08T15:56:07.290 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-05-08T15:56:07.290 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-05-08T15:56:07.290 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-05-08T15:56:07.290 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-05-08T15:56:07.290 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-05-08T15:56:07.290 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-05-08T15:56:07.291 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-05-08T15:56:07.291 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-05-08T15:56:07.291 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-08T15:56:07.292 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T15:56:07.292 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20299 2024-05-08T15:56:07.292 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20299/ceph-mon.a.asok 2024-05-08T15:56:07.292 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-05-08T15:56:07.292 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.20299/ceph-mon.a.asok config get mon_host 2024-05-08T15:56:07.392 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:96: TEST_ec_profile_warning: run_mgr td/test-erasure-code-plugins x 2024-05-08T15:56:07.393 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/test-erasure-code-plugins 2024-05-08T15:56:07.393 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-05-08T15:56:07.393 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-05-08T15:56:07.393 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-05-08T15:56:07.393 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/test-erasure-code-plugins/x 2024-05-08T15:56:07.393 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-05-08T15:56:08.334 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-05-08T15:56:08.334 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-08T15:56:08.334 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-08T15:56:08.334 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-08T15:56:08.334 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T15:56:08.334 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20299 2024-05-08T15:56:08.334 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.20299/$cluster-$name.asok' 2024-05-08T15:56:08.335 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-05-08T15:56:08.347 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-code-plugins/x '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.20299/$cluster-$name.asok' --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-05-08T15:56:08.475 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:97: TEST_ec_profile_warning: seq 0 2 2024-05-08T15:56:08.479 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:97: TEST_ec_profile_warning: for id in $(seq 0 2) 2024-05-08T15:56:08.479 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:98: TEST_ec_profile_warning: run_osd td/test-erasure-code-plugins 0 2024-05-08T15:56:08.480 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-code-plugins 2024-05-08T15:56:08.480 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-05-08T15:56:08.480 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-05-08T15:56:08.480 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-05-08T15:56:08.480 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-code-plugins/0 2024-05-08T15:56:08.480 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=2a3a20f9-46fd-4090-ba60-6d102331c696 --auth-supported=none --mon-host=127.0.0.1:17110 ' 2024-05-08T15:56:08.480 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-08T15:56:08.480 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-05-08T15:56:08.480 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-08T15:56:08.480 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-code-plugins/0' 2024-05-08T15:56:08.480 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code-plugins/0/journal' 2024-05-08T15:56:08.480 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-05-08T15:56:08.481 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-05-08T15:56:08.481 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-code-plugins' 2024-05-08T15:56:08.481 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-05-08T15:56:08.481 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-08T15:56:08.481 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-08T15:56:08.481 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-08T15:56:08.481 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T15:56:08.481 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20299 2024-05-08T15:56:08.481 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.20299/$cluster-$name.asok' 2024-05-08T15:56:08.481 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.20299/$cluster-$name.asok' 2024-05-08T15:56:08.481 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-05-08T15:56:08.481 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-05-08T15:56:08.481 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-05-08T15:56:08.482 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-code-plugins/$name.log' 2024-05-08T15:56:08.482 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-code-plugins/$name.pid' 2024-05-08T15:56:08.482 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-08T15:56:08.482 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-08T15:56:08.482 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-08T15:56:08.482 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-08T15:56:08.482 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-05-08T15:56:08.482 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-05-08T15:56:08.482 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-code-plugins/0 2024-05-08T15:56:08.483 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-05-08T15:56:08.483 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=ca521b58-9b90-4a08-a509-69919107b2c7 2024-05-08T15:56:08.483 INFO:tasks.workunit.client.0.smithi172.stdout:add osd0 ca521b58-9b90-4a08-a509-69919107b2c7 2024-05-08T15:56:08.484 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 ca521b58-9b90-4a08-a509-69919107b2c7' 2024-05-08T15:56:08.484 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-05-08T15:56:08.499 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCYoDtm1U9HHhAABaNv4Zqf/VvvsmQdIv3+8A== 2024-05-08T15:56:08.499 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCYoDtm1U9HHhAABaNv4Zqf/VvvsmQdIv3+8A=="}' 2024-05-08T15:56:08.499 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new ca521b58-9b90-4a08-a509-69919107b2c7 -i td/test-erasure-code-plugins/0/new.json 2024-05-08T15:56:08.821 INFO:tasks.workunit.client.0.smithi172.stdout:0 2024-05-08T15:56:08.837 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-code-plugins/0/new.json 2024-05-08T15:56:08.838 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=2a3a20f9-46fd-4090-ba60-6d102331c696 --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.20299/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCYoDtm1U9HHhAABaNv4Zqf/VvvsmQdIv3+8A== --osd-uuid ca521b58-9b90-4a08-a509-69919107b2c7 2024-05-08T15:56:08.971 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T15:56:08.977+0000 7f3bf37727c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T15:56:08.974 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T15:56:08.981+0000 7f3bf37727c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T15:56:08.975 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T15:56:08.981+0000 7f3bf37727c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T15:56:08.976 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T15:56:08.981+0000 7f3bf37727c0 -1 bluestore(td/test-erasure-code-plugins/0/block) _read_bdev_label failed to open td/test-erasure-code-plugins/0/block: (2) No such file or directory 2024-05-08T15:56:08.976 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T15:56:08.981+0000 7f3bf37727c0 -1 bluestore(td/test-erasure-code-plugins/0/block) _read_bdev_label failed to open td/test-erasure-code-plugins/0/block: (2) No such file or directory 2024-05-08T15:56:08.976 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T15:56:08.981+0000 7f3bf37727c0 -1 bluestore(td/test-erasure-code-plugins/0/block) _read_bdev_label failed to open td/test-erasure-code-plugins/0/block: (2) No such file or directory 2024-05-08T15:56:09.097 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T15:56:09.101+0000 7f3bf37727c0 -1 bluestore(td/test-erasure-code-plugins/0) _read_fsid unparsable uuid 2024-05-08T15:56:15.374 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-code-plugins/0/keyring 2024-05-08T15:56:15.375 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-05-08T15:56:15.376 INFO:tasks.workunit.client.0.smithi172.stdout:adding osd0 key to auth repository 2024-05-08T15:56:15.376 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-05-08T15:56:15.376 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-code-plugins/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-05-08T15:56:15.614 INFO:tasks.workunit.client.0.smithi172.stdout:start osd.0 2024-05-08T15:56:15.615 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-05-08T15:56:15.615 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-05-08T15:56:15.615 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-05-08T15:56:15.615 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=2a3a20f9-46fd-4090-ba60-6d102331c696 --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.20299/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-05-08T15:56:15.615 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-05-08T15:56:15.645 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T15:56:15.649+0000 7fbf900a37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T15:56:15.648 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T15:56:15.653+0000 7fbf900a37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T15:56:15.649 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T15:56:15.653+0000 7fbf900a37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T15:56:15.833 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-05-08T15:56:15.833 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-08T15:56:15.833 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-05-08T15:56:15.833 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-08T15:56:15.833 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-08T15:56:15.833 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T15:56:15.833 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-08T15:56:15.834 INFO:tasks.workunit.client.0.smithi172.stdout:0 2024-05-08T15:56:15.834 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T15:56:15.834 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T15:56:16.061 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T15:56:16.224 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T15:56:16.229+0000 7fbf900a37c0 -1 Falling back to public interface 2024-05-08T15:56:17.062 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T15:56:17.063 INFO:tasks.workunit.client.0.smithi172.stdout:1 2024-05-08T15:56:17.063 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T15:56:17.063 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-08T15:56:17.063 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T15:56:17.063 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T15:56:17.324 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T15:56:18.326 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T15:56:18.326 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T15:56:18.326 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-08T15:56:18.326 INFO:tasks.workunit.client.0.smithi172.stdout:2 2024-05-08T15:56:18.327 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T15:56:18.327 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T15:56:18.575 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T15:56:18.581+0000 7fbf900a37c0 -1 osd.0 0 log_to_monitors true 2024-05-08T15:56:18.780 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T15:56:19.782 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T15:56:19.783 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T15:56:19.783 INFO:tasks.workunit.client.0.smithi172.stdout:3 2024-05-08T15:56:19.783 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-05-08T15:56:19.784 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T15:56:19.784 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T15:56:20.283 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T15:56:21.285 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T15:56:21.285 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T15:56:21.285 INFO:tasks.workunit.client.0.smithi172.stdout:4 2024-05-08T15:56:21.286 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-05-08T15:56:21.286 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T15:56:21.286 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T15:56:21.721 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T15:56:22.723 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T15:56:22.723 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T15:56:22.723 INFO:tasks.workunit.client.0.smithi172.stdout:5 2024-05-08T15:56:22.724 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-05-08T15:56:22.724 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T15:56:22.724 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T15:56:23.149 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T15:56:24.151 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T15:56:24.151 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T15:56:24.151 INFO:tasks.workunit.client.0.smithi172.stdout:6 2024-05-08T15:56:24.151 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-05-08T15:56:24.151 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T15:56:24.152 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T15:56:24.590 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T15:56:25.592 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T15:56:25.592 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T15:56:25.592 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-05-08T15:56:25.593 INFO:tasks.workunit.client.0.smithi172.stdout:7 2024-05-08T15:56:25.593 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T15:56:25.593 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T15:56:26.034 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T15:56:27.036 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T15:56:27.036 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T15:56:27.036 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-05-08T15:56:27.037 INFO:tasks.workunit.client.0.smithi172.stdout:8 2024-05-08T15:56:27.037 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T15:56:27.037 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T15:56:27.560 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T15:56:28.562 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T15:56:28.562 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T15:56:28.562 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-05-08T15:56:28.562 INFO:tasks.workunit.client.0.smithi172.stdout:9 2024-05-08T15:56:28.563 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T15:56:28.563 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T15:56:29.076 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T15:56:30.078 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T15:56:30.079 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T15:56:30.079 INFO:tasks.workunit.client.0.smithi172.stdout:10 2024-05-08T15:56:30.079 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-05-08T15:56:30.080 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T15:56:30.080 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T15:56:30.659 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T15:56:31.662 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T15:56:31.662 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T15:56:31.662 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-05-08T15:56:31.662 INFO:tasks.workunit.client.0.smithi172.stdout:11 2024-05-08T15:56:31.663 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T15:56:31.663 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T15:56:32.082 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T15:56:33.084 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T15:56:33.084 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T15:56:33.084 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-05-08T15:56:33.084 INFO:tasks.workunit.client.0.smithi172.stdout:12 2024-05-08T15:56:33.085 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T15:56:33.085 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T15:56:34.181 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T15:56:35.183 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T15:56:35.183 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T15:56:35.183 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-05-08T15:56:35.184 INFO:tasks.workunit.client.0.smithi172.stdout:13 2024-05-08T15:56:35.184 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T15:56:35.184 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T15:56:35.794 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T15:56:36.796 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T15:56:36.796 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T15:56:36.796 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-05-08T15:56:36.796 INFO:tasks.workunit.client.0.smithi172.stdout:14 2024-05-08T15:56:36.797 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T15:56:36.797 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T15:56:37.237 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T15:56:38.239 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T15:56:38.239 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T15:56:38.239 INFO:tasks.workunit.client.0.smithi172.stdout:15 2024-05-08T15:56:38.240 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-05-08T15:56:38.240 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T15:56:38.240 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T15:56:39.288 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T15:56:40.289 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T15:56:40.289 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T15:56:40.289 INFO:tasks.workunit.client.0.smithi172.stdout:16 2024-05-08T15:56:40.290 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-05-08T15:56:40.290 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T15:56:40.290 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T15:56:41.361 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T15:56:42.363 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T15:56:42.363 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T15:56:42.363 INFO:tasks.workunit.client.0.smithi172.stdout:17 2024-05-08T15:56:42.364 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-05-08T15:56:42.364 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T15:56:42.364 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T15:56:43.649 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T15:56:44.651 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T15:56:44.652 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T15:56:44.652 INFO:tasks.workunit.client.0.smithi172.stdout:18 2024-05-08T15:56:44.653 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-05-08T15:56:44.653 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T15:56:44.654 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T15:56:45.254 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T15:56:46.255 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T15:56:46.255 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T15:56:46.256 INFO:tasks.workunit.client.0.smithi172.stdout:19 2024-05-08T15:56:46.256 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-05-08T15:56:46.256 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T15:56:46.256 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T15:56:46.685 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T15:56:47.686 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T15:56:47.686 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T15:56:47.686 INFO:tasks.workunit.client.0.smithi172.stdout:20 2024-05-08T15:56:47.687 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-05-08T15:56:47.687 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T15:56:47.687 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T15:56:48.108 INFO:tasks.workunit.client.0.smithi172.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/1456498050,v1:127.0.0.1:6801/1456498050] [v2:127.0.0.1:6802/1456498050,v1:127.0.0.1:6803/1456498050] exists,up ca521b58-9b90-4a08-a509-69919107b2c7 2024-05-08T15:56:48.113 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-08T15:56:48.113 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-08T15:56:48.114 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-08T15:56:48.114 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:97: TEST_ec_profile_warning: for id in $(seq 0 2) 2024-05-08T15:56:48.114 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:98: TEST_ec_profile_warning: run_osd td/test-erasure-code-plugins 1 2024-05-08T15:56:48.114 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-code-plugins 2024-05-08T15:56:48.114 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-05-08T15:56:48.114 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-05-08T15:56:48.114 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-05-08T15:56:48.114 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-code-plugins/1 2024-05-08T15:56:48.115 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=2a3a20f9-46fd-4090-ba60-6d102331c696 --auth-supported=none --mon-host=127.0.0.1:17110 ' 2024-05-08T15:56:48.115 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-08T15:56:48.115 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-05-08T15:56:48.115 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-08T15:56:48.115 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-code-plugins/1' 2024-05-08T15:56:48.115 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code-plugins/1/journal' 2024-05-08T15:56:48.115 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-05-08T15:56:48.115 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-05-08T15:56:48.115 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-code-plugins' 2024-05-08T15:56:48.115 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-05-08T15:56:48.115 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-08T15:56:48.116 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-08T15:56:48.116 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-08T15:56:48.116 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T15:56:48.116 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20299 2024-05-08T15:56:48.116 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.20299/$cluster-$name.asok' 2024-05-08T15:56:48.116 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.20299/$cluster-$name.asok' 2024-05-08T15:56:48.116 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-05-08T15:56:48.116 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-05-08T15:56:48.116 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-05-08T15:56:48.116 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-code-plugins/$name.log' 2024-05-08T15:56:48.117 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-code-plugins/$name.pid' 2024-05-08T15:56:48.117 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-08T15:56:48.117 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-08T15:56:48.117 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-08T15:56:48.117 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-08T15:56:48.117 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-05-08T15:56:48.117 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-05-08T15:56:48.117 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-code-plugins/1 2024-05-08T15:56:48.117 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-05-08T15:56:48.117 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=a8991bb7-a79e-4a6b-a8e6-4a130913444d 2024-05-08T15:56:48.118 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 a8991bb7-a79e-4a6b-a8e6-4a130913444d' 2024-05-08T15:56:48.118 INFO:tasks.workunit.client.0.smithi172.stdout:add osd1 a8991bb7-a79e-4a6b-a8e6-4a130913444d 2024-05-08T15:56:48.118 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-05-08T15:56:48.128 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDAoDtm0wkGCBAAh4iLiyxRDR3d+ouYbEtf6g== 2024-05-08T15:56:48.128 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDAoDtm0wkGCBAAh4iLiyxRDR3d+ouYbEtf6g=="}' 2024-05-08T15:56:48.128 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new a8991bb7-a79e-4a6b-a8e6-4a130913444d -i td/test-erasure-code-plugins/1/new.json 2024-05-08T15:56:48.602 INFO:tasks.workunit.client.0.smithi172.stdout:1 2024-05-08T15:56:48.618 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-code-plugins/1/new.json 2024-05-08T15:56:48.619 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=2a3a20f9-46fd-4090-ba60-6d102331c696 --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/1 --osd-journal=td/test-erasure-code-plugins/1/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.20299/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDAoDtm0wkGCBAAh4iLiyxRDR3d+ouYbEtf6g== --osd-uuid a8991bb7-a79e-4a6b-a8e6-4a130913444d 2024-05-08T15:56:48.650 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T15:56:48.654+0000 7f5b1796d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T15:56:48.682 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T15:56:48.686+0000 7f5b1796d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T15:56:48.684 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T15:56:48.690+0000 7f5b1796d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T15:56:48.684 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T15:56:48.690+0000 7f5b1796d7c0 -1 bluestore(td/test-erasure-code-plugins/1/block) _read_bdev_label failed to open td/test-erasure-code-plugins/1/block: (2) No such file or directory 2024-05-08T15:56:48.684 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T15:56:48.690+0000 7f5b1796d7c0 -1 bluestore(td/test-erasure-code-plugins/1/block) _read_bdev_label failed to open td/test-erasure-code-plugins/1/block: (2) No such file or directory 2024-05-08T15:56:48.685 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T15:56:48.690+0000 7f5b1796d7c0 -1 bluestore(td/test-erasure-code-plugins/1/block) _read_bdev_label failed to open td/test-erasure-code-plugins/1/block: (2) No such file or directory 2024-05-08T15:56:48.722 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T15:56:48.726+0000 7f5b1796d7c0 -1 bluestore(td/test-erasure-code-plugins/1) _read_fsid unparsable uuid 2024-05-08T15:56:55.365 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-code-plugins/1/keyring 2024-05-08T15:56:55.365 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-05-08T15:56:55.366 INFO:tasks.workunit.client.0.smithi172.stdout:adding osd1 key to auth repository 2024-05-08T15:56:55.366 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-05-08T15:56:55.366 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-code-plugins/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-05-08T15:56:55.918 INFO:tasks.workunit.client.0.smithi172.stdout:start osd.1 2024-05-08T15:56:55.919 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-05-08T15:56:55.919 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=2a3a20f9-46fd-4090-ba60-6d102331c696 --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/1 --osd-journal=td/test-erasure-code-plugins/1/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.20299/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-05-08T15:56:55.920 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-05-08T15:56:55.920 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-05-08T15:56:55.920 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-05-08T15:56:55.946 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T15:56:55.949+0000 7f41b9fb47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T15:56:55.948 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T15:56:55.953+0000 7f41b9fb47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T15:56:55.950 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T15:56:55.953+0000 7f41b9fb47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T15:56:56.301 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-05-08T15:56:56.302 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-08T15:56:56.302 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-05-08T15:56:56.302 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-08T15:56:56.302 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-08T15:56:56.302 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T15:56:56.302 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-08T15:56:56.302 INFO:tasks.workunit.client.0.smithi172.stdout:0 2024-05-08T15:56:56.303 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T15:56:56.303 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T15:56:56.486 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T15:56:56.489+0000 7f41b9fb47c0 -1 Falling back to public interface 2024-05-08T15:56:56.718 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T15:56:57.720 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T15:56:57.720 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T15:56:57.720 INFO:tasks.workunit.client.0.smithi172.stdout:1 2024-05-08T15:56:57.721 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-08T15:56:57.721 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T15:56:57.721 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T15:56:58.126 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T15:56:59.090 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T15:56:59.093+0000 7f41b9fb47c0 -1 osd.1 0 log_to_monitors true 2024-05-08T15:56:59.127 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T15:56:59.127 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T15:56:59.127 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-08T15:56:59.127 INFO:tasks.workunit.client.0.smithi172.stdout:2 2024-05-08T15:56:59.128 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T15:56:59.128 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T15:56:59.524 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T15:57:00.525 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T15:57:00.525 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T15:57:00.526 INFO:tasks.workunit.client.0.smithi172.stdout:3 2024-05-08T15:57:00.526 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-05-08T15:57:00.526 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T15:57:00.526 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T15:57:01.015 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T15:57:02.018 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T15:57:02.018 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T15:57:02.018 INFO:tasks.workunit.client.0.smithi172.stdout:4 2024-05-08T15:57:02.019 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-05-08T15:57:02.019 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T15:57:02.019 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T15:57:02.565 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T15:57:03.567 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T15:57:03.567 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T15:57:03.567 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-05-08T15:57:03.568 INFO:tasks.workunit.client.0.smithi172.stdout:5 2024-05-08T15:57:03.568 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T15:57:03.568 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T15:57:04.406 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T15:57:05.408 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T15:57:05.408 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T15:57:05.408 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-05-08T15:57:05.409 INFO:tasks.workunit.client.0.smithi172.stdout:6 2024-05-08T15:57:05.409 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T15:57:05.409 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T15:57:05.974 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T15:57:06.976 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T15:57:06.976 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T15:57:06.976 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-05-08T15:57:06.976 INFO:tasks.workunit.client.0.smithi172.stdout:7 2024-05-08T15:57:06.977 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T15:57:06.977 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T15:57:07.463 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T15:57:08.464 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T15:57:08.465 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T15:57:08.465 INFO:tasks.workunit.client.0.smithi172.stdout:8 2024-05-08T15:57:08.465 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-05-08T15:57:08.465 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T15:57:08.466 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T15:57:09.101 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T15:57:10.103 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T15:57:10.103 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T15:57:10.104 INFO:tasks.workunit.client.0.smithi172.stdout:9 2024-05-08T15:57:10.104 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-05-08T15:57:10.104 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T15:57:10.104 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T15:57:10.527 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T15:57:11.529 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T15:57:11.529 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T15:57:11.529 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-05-08T15:57:11.529 INFO:tasks.workunit.client.0.smithi172.stdout:10 2024-05-08T15:57:11.530 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T15:57:11.530 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T15:57:11.954 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T15:57:12.956 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T15:57:12.956 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T15:57:12.956 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-05-08T15:57:12.957 INFO:tasks.workunit.client.0.smithi172.stdout:11 2024-05-08T15:57:12.957 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T15:57:12.957 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T15:57:13.364 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T15:57:14.365 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T15:57:14.366 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T15:57:14.366 INFO:tasks.workunit.client.0.smithi172.stdout:12 2024-05-08T15:57:14.366 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-05-08T15:57:14.367 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T15:57:14.367 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T15:57:15.034 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T15:57:16.035 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T15:57:16.035 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T15:57:16.036 INFO:tasks.workunit.client.0.smithi172.stdout:13 2024-05-08T15:57:16.036 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-05-08T15:57:16.036 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T15:57:16.036 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T15:57:16.729 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T15:57:17.730 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T15:57:17.731 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T15:57:17.731 INFO:tasks.workunit.client.0.smithi172.stdout:14 2024-05-08T15:57:17.731 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-05-08T15:57:17.731 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T15:57:17.731 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T15:57:18.690 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T15:57:19.692 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T15:57:19.692 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T15:57:19.692 INFO:tasks.workunit.client.0.smithi172.stdout:15 2024-05-08T15:57:19.693 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-05-08T15:57:19.693 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T15:57:19.693 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T15:57:20.627 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T15:57:21.629 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T15:57:21.629 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T15:57:21.629 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-05-08T15:57:21.629 INFO:tasks.workunit.client.0.smithi172.stdout:16 2024-05-08T15:57:21.629 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T15:57:21.630 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T15:57:22.719 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T15:57:23.721 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T15:57:23.721 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T15:57:23.721 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-05-08T15:57:23.722 INFO:tasks.workunit.client.0.smithi172.stdout:17 2024-05-08T15:57:23.722 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T15:57:23.722 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T15:57:24.616 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T15:57:25.618 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T15:57:25.618 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T15:57:25.618 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-05-08T15:57:25.618 INFO:tasks.workunit.client.0.smithi172.stdout:18 2024-05-08T15:57:25.619 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T15:57:25.619 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T15:57:26.078 INFO:tasks.workunit.client.0.smithi172.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/607851076,v1:127.0.0.1:6811/607851076] [v2:127.0.0.1:6812/607851076,v1:127.0.0.1:6813/607851076] exists,up a8991bb7-a79e-4a6b-a8e6-4a130913444d 2024-05-08T15:57:26.080 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-08T15:57:26.080 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-08T15:57:26.080 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-08T15:57:26.080 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:97: TEST_ec_profile_warning: for id in $(seq 0 2) 2024-05-08T15:57:26.080 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:98: TEST_ec_profile_warning: run_osd td/test-erasure-code-plugins 2 2024-05-08T15:57:26.080 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-code-plugins 2024-05-08T15:57:26.081 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-05-08T15:57:26.081 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-05-08T15:57:26.081 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-05-08T15:57:26.081 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-code-plugins/2 2024-05-08T15:57:26.081 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=2a3a20f9-46fd-4090-ba60-6d102331c696 --auth-supported=none --mon-host=127.0.0.1:17110 ' 2024-05-08T15:57:26.081 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-08T15:57:26.081 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-05-08T15:57:26.081 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-08T15:57:26.081 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-code-plugins/2' 2024-05-08T15:57:26.081 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code-plugins/2/journal' 2024-05-08T15:57:26.081 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-05-08T15:57:26.081 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-05-08T15:57:26.081 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-code-plugins' 2024-05-08T15:57:26.081 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-05-08T15:57:26.082 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-08T15:57:26.082 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-08T15:57:26.082 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-08T15:57:26.082 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T15:57:26.082 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20299 2024-05-08T15:57:26.083 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.20299/$cluster-$name.asok' 2024-05-08T15:57:26.084 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.20299/$cluster-$name.asok' 2024-05-08T15:57:26.084 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-05-08T15:57:26.084 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-05-08T15:57:26.084 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-05-08T15:57:26.084 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-code-plugins/$name.log' 2024-05-08T15:57:26.084 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-code-plugins/$name.pid' 2024-05-08T15:57:26.084 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-08T15:57:26.084 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-08T15:57:26.084 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-08T15:57:26.084 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-08T15:57:26.084 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-05-08T15:57:26.084 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-05-08T15:57:26.084 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-code-plugins/2 2024-05-08T15:57:26.084 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-05-08T15:57:26.085 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=d4bfb29f-5c80-40a3-a52c-3d656629a77b 2024-05-08T15:57:26.085 INFO:tasks.workunit.client.0.smithi172.stdout:add osd2 d4bfb29f-5c80-40a3-a52c-3d656629a77b 2024-05-08T15:57:26.085 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 d4bfb29f-5c80-40a3-a52c-3d656629a77b' 2024-05-08T15:57:26.085 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-05-08T15:57:26.105 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDmoDtm412SBhAAy+Kjyr6iW7FTMbw0dMvopg== 2024-05-08T15:57:26.106 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDmoDtm412SBhAAy+Kjyr6iW7FTMbw0dMvopg=="}' 2024-05-08T15:57:26.106 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new d4bfb29f-5c80-40a3-a52c-3d656629a77b -i td/test-erasure-code-plugins/2/new.json 2024-05-08T15:57:26.586 INFO:tasks.workunit.client.0.smithi172.stdout:2 2024-05-08T15:57:26.602 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-code-plugins/2/new.json 2024-05-08T15:57:26.603 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=2a3a20f9-46fd-4090-ba60-6d102331c696 --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/2 --osd-journal=td/test-erasure-code-plugins/2/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.20299/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDmoDtm412SBhAAy+Kjyr6iW7FTMbw0dMvopg== --osd-uuid d4bfb29f-5c80-40a3-a52c-3d656629a77b 2024-05-08T15:57:26.635 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T15:57:26.636+0000 7fccef80b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T15:57:26.645 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T15:57:26.648+0000 7fccef80b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T15:57:26.647 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T15:57:26.648+0000 7fccef80b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T15:57:26.647 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T15:57:26.648+0000 7fccef80b7c0 -1 bluestore(td/test-erasure-code-plugins/2/block) _read_bdev_label failed to open td/test-erasure-code-plugins/2/block: (2) No such file or directory 2024-05-08T15:57:26.647 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T15:57:26.648+0000 7fccef80b7c0 -1 bluestore(td/test-erasure-code-plugins/2/block) _read_bdev_label failed to open td/test-erasure-code-plugins/2/block: (2) No such file or directory 2024-05-08T15:57:26.647 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T15:57:26.648+0000 7fccef80b7c0 -1 bluestore(td/test-erasure-code-plugins/2/block) _read_bdev_label failed to open td/test-erasure-code-plugins/2/block: (2) No such file or directory 2024-05-08T15:57:26.685 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T15:57:26.688+0000 7fccef80b7c0 -1 bluestore(td/test-erasure-code-plugins/2) _read_fsid unparsable uuid 2024-05-08T15:57:33.354 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-code-plugins/2/keyring 2024-05-08T15:57:33.355 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-05-08T15:57:33.355 INFO:tasks.workunit.client.0.smithi172.stdout:adding osd2 key to auth repository 2024-05-08T15:57:33.356 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-05-08T15:57:33.356 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-code-plugins/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-05-08T15:57:33.890 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-05-08T15:57:33.890 INFO:tasks.workunit.client.0.smithi172.stdout:start osd.2 2024-05-08T15:57:33.891 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=2a3a20f9-46fd-4090-ba60-6d102331c696 --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/2 --osd-journal=td/test-erasure-code-plugins/2/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.20299/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-05-08T15:57:33.891 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-05-08T15:57:33.891 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-05-08T15:57:33.891 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-05-08T15:57:33.921 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T15:57:33.920+0000 7f7020cb87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T15:57:33.922 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T15:57:33.924+0000 7f7020cb87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T15:57:33.923 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T15:57:33.924+0000 7f7020cb87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T15:57:34.332 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-05-08T15:57:34.332 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-08T15:57:34.332 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-05-08T15:57:34.332 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-08T15:57:34.332 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-08T15:57:34.333 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T15:57:34.333 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-08T15:57:34.333 INFO:tasks.workunit.client.0.smithi172.stdout:0 2024-05-08T15:57:34.333 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T15:57:34.333 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T15:57:34.495 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T15:57:34.496+0000 7f7020cb87c0 -1 Falling back to public interface 2024-05-08T15:57:34.726 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T15:57:35.728 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T15:57:35.728 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T15:57:35.728 INFO:tasks.workunit.client.0.smithi172.stdout:1 2024-05-08T15:57:35.729 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-08T15:57:35.729 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T15:57:35.729 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T15:57:36.141 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T15:57:37.064 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T15:57:37.063+0000 7f7020cb87c0 -1 osd.2 0 log_to_monitors true 2024-05-08T15:57:37.142 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T15:57:37.143 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T15:57:37.143 INFO:tasks.workunit.client.0.smithi172.stdout:2 2024-05-08T15:57:37.143 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-08T15:57:37.143 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T15:57:37.143 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T15:57:37.833 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T15:57:38.834 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T15:57:38.834 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T15:57:38.835 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-05-08T15:57:38.835 INFO:tasks.workunit.client.0.smithi172.stdout:3 2024-05-08T15:57:38.835 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T15:57:38.835 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T15:57:39.337 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T15:57:40.338 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T15:57:40.339 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T15:57:40.339 INFO:tasks.workunit.client.0.smithi172.stdout:4 2024-05-08T15:57:40.339 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-05-08T15:57:40.339 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T15:57:40.340 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T15:57:40.758 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T15:57:41.759 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T15:57:41.759 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T15:57:41.760 INFO:tasks.workunit.client.0.smithi172.stdout:5 2024-05-08T15:57:41.760 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-05-08T15:57:41.760 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T15:57:41.760 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T15:57:42.260 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T15:57:43.261 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T15:57:43.261 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T15:57:43.262 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-05-08T15:57:43.262 INFO:tasks.workunit.client.0.smithi172.stdout:6 2024-05-08T15:57:43.262 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T15:57:43.262 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T15:57:43.847 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T15:57:44.849 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T15:57:44.849 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T15:57:44.849 INFO:tasks.workunit.client.0.smithi172.stdout:7 2024-05-08T15:57:44.850 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-05-08T15:57:44.850 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T15:57:44.850 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T15:57:45.338 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T15:57:46.340 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T15:57:46.340 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T15:57:46.340 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-05-08T15:57:46.340 INFO:tasks.workunit.client.0.smithi172.stdout:8 2024-05-08T15:57:46.341 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T15:57:46.341 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T15:57:46.770 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T15:57:47.771 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T15:57:47.771 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T15:57:47.772 INFO:tasks.workunit.client.0.smithi172.stdout:9 2024-05-08T15:57:47.772 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-05-08T15:57:47.772 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T15:57:47.772 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T15:57:48.307 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T15:57:49.309 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T15:57:49.309 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T15:57:49.309 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-05-08T15:57:49.309 INFO:tasks.workunit.client.0.smithi172.stdout:10 2024-05-08T15:57:49.310 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T15:57:49.310 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T15:57:49.814 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T15:57:50.816 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T15:57:50.816 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T15:57:50.816 INFO:tasks.workunit.client.0.smithi172.stdout:11 2024-05-08T15:57:50.817 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-05-08T15:57:50.817 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T15:57:50.817 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T15:57:51.690 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T15:57:52.691 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T15:57:52.691 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T15:57:52.692 INFO:tasks.workunit.client.0.smithi172.stdout:12 2024-05-08T15:57:52.692 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-05-08T15:57:52.692 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T15:57:52.692 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T15:57:53.463 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T15:57:54.464 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T15:57:54.465 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T15:57:54.465 INFO:tasks.workunit.client.0.smithi172.stdout:13 2024-05-08T15:57:54.465 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-05-08T15:57:54.466 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T15:57:54.466 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T15:57:55.290 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T15:57:56.291 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T15:57:56.291 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T15:57:56.292 INFO:tasks.workunit.client.0.smithi172.stdout:14 2024-05-08T15:57:56.292 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-05-08T15:57:56.292 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T15:57:56.292 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T15:57:56.937 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T15:57:57.938 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T15:57:57.939 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T15:57:57.939 INFO:tasks.workunit.client.0.smithi172.stdout:15 2024-05-08T15:57:57.939 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-05-08T15:57:57.940 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T15:57:57.940 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T15:57:59.040 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T15:58:00.042 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T15:58:00.042 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T15:58:00.042 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-05-08T15:58:00.042 INFO:tasks.workunit.client.0.smithi172.stdout:16 2024-05-08T15:58:00.043 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T15:58:00.043 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T15:58:00.973 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T15:58:01.975 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T15:58:01.975 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T15:58:01.975 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-05-08T15:58:01.975 INFO:tasks.workunit.client.0.smithi172.stdout:17 2024-05-08T15:58:01.976 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T15:58:01.976 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T15:58:02.442 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T15:58:03.444 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T15:58:03.445 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T15:58:03.445 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-05-08T15:58:03.445 INFO:tasks.workunit.client.0.smithi172.stdout:18 2024-05-08T15:58:03.446 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T15:58:03.446 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T15:58:03.871 INFO:tasks.workunit.client.0.smithi172.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/1529245556,v1:127.0.0.1:6819/1529245556] [v2:127.0.0.1:6820/1529245556,v1:127.0.0.1:6821/1529245556] exists,up d4bfb29f-5c80-40a3-a52c-3d656629a77b 2024-05-08T15:58:03.872 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-08T15:58:03.872 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-08T15:58:03.872 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-08T15:58:03.872 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:100: TEST_ec_profile_warning: create_rbd_pool 2024-05-08T15:58:03.872 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-05-08T15:58:04.309 INFO:tasks.workunit.client.0.smithi172.stderr:pool 'rbd' does not exist 2024-05-08T15:58:04.324 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-05-08T15:58:04.325 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-05-08T15:58:04.878 INFO:tasks.workunit.client.0.smithi172.stderr:pool 'rbd' already exists 2024-05-08T15:58:04.893 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-05-08T15:58:05.896 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-05-08T15:58:08.154 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:101: TEST_ec_profile_warning: wait_for_clean 2024-05-08T15:58:08.154 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-05-08T15:58:08.154 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-05-08T15:58:08.154 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-05-08T15:58:08.155 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-05-08T15:58:08.156 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-05-08T15:58:08.156 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-05-08T15:58:08.156 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-05-08T15:58:08.156 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-05-08T15:58:08.157 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-05-08T15:58:08.277 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-05-08T15:58:08.278 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-05-08T15:58:08.278 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-05-08T15:58:08.278 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-05-08T15:58:08.278 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-05-08T15:58:08.278 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-05-08T15:58:08.698 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-05-08T15:58:08.699 INFO:tasks.workunit.client.0.smithi172.stderr:1 2024-05-08T15:58:08.699 INFO:tasks.workunit.client.0.smithi172.stderr:2' 2024-05-08T15:58:08.699 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-05-08T15:58:08.699 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T15:58:08.699 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-05-08T15:58:08.884 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836500 2024-05-08T15:58:08.885 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836500 2024-05-08T15:58:08.885 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836500' 2024-05-08T15:58:08.885 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T15:58:08.885 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-05-08T15:58:09.034 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949672980 2024-05-08T15:58:09.035 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949672980 2024-05-08T15:58:09.035 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836500 1-42949672980' 2024-05-08T15:58:09.035 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T15:58:09.035 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-05-08T15:58:09.182 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509447 2024-05-08T15:58:09.183 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509447 2024-05-08T15:58:09.183 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836500 1-42949672980 2-64424509447' 2024-05-08T15:58:09.183 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-08T15:58:09.183 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836500 2024-05-08T15:58:09.183 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-08T15:58:09.184 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-05-08T15:58:09.185 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836500 2024-05-08T15:58:09.185 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-08T15:58:09.186 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836500 2024-05-08T15:58:09.186 INFO:tasks.workunit.client.0.smithi172.stdout:waiting osd.0 seq 21474836500 2024-05-08T15:58:09.187 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836500' 2024-05-08T15:58:09.187 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-05-08T15:58:09.587 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836500 -lt 21474836500 2024-05-08T15:58:09.588 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-08T15:58:09.589 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949672980 2024-05-08T15:58:09.589 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-08T15:58:09.590 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-05-08T15:58:09.591 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949672980 2024-05-08T15:58:09.591 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-08T15:58:09.592 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949672980 2024-05-08T15:58:09.592 INFO:tasks.workunit.client.0.smithi172.stdout:waiting osd.1 seq 42949672980 2024-05-08T15:58:09.593 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949672980' 2024-05-08T15:58:09.593 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-05-08T15:58:10.014 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672979 -lt 42949672980 2024-05-08T15:58:10.014 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-05-08T15:58:11.015 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-05-08T15:58:11.016 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-05-08T15:58:11.410 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949672980 -lt 42949672980 2024-05-08T15:58:11.411 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-08T15:58:11.411 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509447 2024-05-08T15:58:11.411 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-08T15:58:11.413 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-05-08T15:58:11.414 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509447 2024-05-08T15:58:11.415 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-08T15:58:11.415 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509447 2024-05-08T15:58:11.415 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509447' 2024-05-08T15:58:11.415 INFO:tasks.workunit.client.0.smithi172.stdout:waiting osd.2 seq 64424509447 2024-05-08T15:58:11.416 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-05-08T15:58:11.835 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509447 -lt 64424509447 2024-05-08T15:58:11.836 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-05-08T15:58:11.837 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-05-08T15:58:11.837 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-05-08T15:58:12.375 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-05-08T15:58:12.376 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-05-08T15:58:12.376 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-05-08T15:58:12.376 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-05-08T15:58:12.376 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-05-08T15:58:12.376 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-05-08T15:58:12.376 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-05-08T15:58:12.376 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-05-08T15:58:12.779 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-05-08T15:58:12.780 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-05-08T15:58:12.781 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-05-08T15:58:12.781 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-05-08T15:58:13.291 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-05-08T15:58:13.292 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-05-08T15:58:13.292 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-05-08T15:58:13.292 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:103: TEST_ec_profile_warning: for plugin in ${legacy_jerasure_plugins[*]} 2024-05-08T15:58:13.292 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:104: TEST_ec_profile_warning: ceph osd erasure-code-profile set prof-jerasure_generic crush-failure-domain=osd technique=reed_sol_van plugin=jerasure_generic 2024-05-08T15:58:13.856 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:105: TEST_ec_profile_warning: get_asok_path mon.a 2024-05-08T15:58:13.856 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-05-08T15:58:13.856 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-05-08T15:58:13.856 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-08T15:58:13.856 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T15:58:13.857 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20299 2024-05-08T15:58:13.857 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20299/ceph-mon.a.asok 2024-05-08T15:58:13.857 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:105: TEST_ec_profile_warning: CEPH_ARGS= 2024-05-08T15:58:13.857 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:105: TEST_ec_profile_warning: ceph --admin-daemon /tmp/ceph-asok.20299/ceph-mon.a.asok log flush 2024-05-08T15:58:13.959 INFO:tasks.workunit.client.0.smithi172.stdout:{} 2024-05-08T15:58:13.971 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:106: TEST_ec_profile_warning: grep 'WARNING: erasure coding profile prof-jerasure_generic uses plugin jerasure_generic' td/test-erasure-code-plugins/mon.a.log 2024-05-08T15:58:13.980 INFO:tasks.workunit.client.0.smithi172.stdout:2024-05-08T15:58:13.594+0000 7f360705c640 0 mon.a@0(leader).osd e23 WARNING: erasure coding profile prof-jerasure_generic uses plugin jerasure_generic that has been deprecated. Please use jerasure instead. 2024-05-08T15:58:13.980 INFO:tasks.workunit.client.0.smithi172.stdout:2024-05-08T15:58:13.838+0000 7f360705c640 0 mon.a@0(leader).osd e24 WARNING: erasure coding profile prof-jerasure_generic uses plugin jerasure_generic that has been deprecated. Please use jerasure instead. 2024-05-08T15:58:13.980 INFO:tasks.workunit.client.0.smithi172.stdout:2024-05-08T15:58:13.838+0000 7f360705c640 0 mon.a@0(leader).osd e24 WARNING: erasure coding profile prof-jerasure_generic uses plugin jerasure_generic that has been deprecated. Please use jerasure instead. 2024-05-08T15:58:13.980 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:103: TEST_ec_profile_warning: for plugin in ${legacy_jerasure_plugins[*]} 2024-05-08T15:58:13.980 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:104: TEST_ec_profile_warning: ceph osd erasure-code-profile set prof-jerasure_sse3 crush-failure-domain=osd technique=reed_sol_van plugin=jerasure_sse3 2024-05-08T15:58:14.526 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:105: TEST_ec_profile_warning: get_asok_path mon.a 2024-05-08T15:58:14.526 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-05-08T15:58:14.526 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-05-08T15:58:14.526 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-08T15:58:14.526 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T15:58:14.526 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20299 2024-05-08T15:58:14.526 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20299/ceph-mon.a.asok 2024-05-08T15:58:14.527 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:105: TEST_ec_profile_warning: CEPH_ARGS= 2024-05-08T15:58:14.527 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:105: TEST_ec_profile_warning: ceph --admin-daemon /tmp/ceph-asok.20299/ceph-mon.a.asok log flush 2024-05-08T15:58:14.614 INFO:tasks.workunit.client.0.smithi172.stdout:{} 2024-05-08T15:58:14.625 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:106: TEST_ec_profile_warning: grep 'WARNING: erasure coding profile prof-jerasure_sse3 uses plugin jerasure_sse3' td/test-erasure-code-plugins/mon.a.log 2024-05-08T15:58:14.633 INFO:tasks.workunit.client.0.smithi172.stdout:2024-05-08T15:58:14.290+0000 7f360705c640 0 mon.a@0(leader).osd e24 WARNING: erasure coding profile prof-jerasure_sse3 uses plugin jerasure_sse3 that has been deprecated. Please use jerasure instead. 2024-05-08T15:58:14.633 INFO:tasks.workunit.client.0.smithi172.stdout:2024-05-08T15:58:14.510+0000 7f360705c640 0 mon.a@0(leader).osd e25 WARNING: erasure coding profile prof-jerasure_sse3 uses plugin jerasure_sse3 that has been deprecated. Please use jerasure instead. 2024-05-08T15:58:14.634 INFO:tasks.workunit.client.0.smithi172.stdout:2024-05-08T15:58:14.510+0000 7f360705c640 0 mon.a@0(leader).osd e25 WARNING: erasure coding profile prof-jerasure_sse3 uses plugin jerasure_sse3 that has been deprecated. Please use jerasure instead. 2024-05-08T15:58:14.634 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:103: TEST_ec_profile_warning: for plugin in ${legacy_jerasure_plugins[*]} 2024-05-08T15:58:14.634 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:104: TEST_ec_profile_warning: ceph osd erasure-code-profile set prof-jerasure_sse4 crush-failure-domain=osd technique=reed_sol_van plugin=jerasure_sse4 2024-05-08T15:58:15.149 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:105: TEST_ec_profile_warning: get_asok_path mon.a 2024-05-08T15:58:15.150 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-05-08T15:58:15.150 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-05-08T15:58:15.150 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-08T15:58:15.150 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T15:58:15.150 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20299 2024-05-08T15:58:15.150 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20299/ceph-mon.a.asok 2024-05-08T15:58:15.150 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:105: TEST_ec_profile_warning: CEPH_ARGS= 2024-05-08T15:58:15.151 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:105: TEST_ec_profile_warning: ceph --admin-daemon /tmp/ceph-asok.20299/ceph-mon.a.asok log flush 2024-05-08T15:58:15.237 INFO:tasks.workunit.client.0.smithi172.stdout:{} 2024-05-08T15:58:15.249 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:106: TEST_ec_profile_warning: grep 'WARNING: erasure coding profile prof-jerasure_sse4 uses plugin jerasure_sse4' td/test-erasure-code-plugins/mon.a.log 2024-05-08T15:58:15.256 INFO:tasks.workunit.client.0.smithi172.stdout:2024-05-08T15:58:14.926+0000 7f360705c640 0 mon.a@0(leader).osd e25 WARNING: erasure coding profile prof-jerasure_sse4 uses plugin jerasure_sse4 that has been deprecated. Please use jerasure instead. 2024-05-08T15:58:15.256 INFO:tasks.workunit.client.0.smithi172.stdout:2024-05-08T15:58:15.134+0000 7f360705c640 0 mon.a@0(leader).osd e26 WARNING: erasure coding profile prof-jerasure_sse4 uses plugin jerasure_sse4 that has been deprecated. Please use jerasure instead. 2024-05-08T15:58:15.257 INFO:tasks.workunit.client.0.smithi172.stdout:2024-05-08T15:58:15.134+0000 7f360705c640 0 mon.a@0(leader).osd e26 WARNING: erasure coding profile prof-jerasure_sse4 uses plugin jerasure_sse4 that has been deprecated. Please use jerasure instead. 2024-05-08T15:58:15.257 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:109: TEST_ec_profile_warning: for plugin in ${legacy_shec_plugins[*]} 2024-05-08T15:58:15.257 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:110: TEST_ec_profile_warning: ceph osd erasure-code-profile set prof-shec_generic crush-failure-domain=osd plugin=shec_generic 2024-05-08T15:58:15.964 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:111: TEST_ec_profile_warning: get_asok_path mon.a 2024-05-08T15:58:15.964 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-05-08T15:58:15.965 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-05-08T15:58:15.965 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-08T15:58:15.965 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T15:58:15.965 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20299 2024-05-08T15:58:15.965 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20299/ceph-mon.a.asok 2024-05-08T15:58:15.966 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:111: TEST_ec_profile_warning: CEPH_ARGS= 2024-05-08T15:58:15.966 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:111: TEST_ec_profile_warning: ceph --admin-daemon /tmp/ceph-asok.20299/ceph-mon.a.asok log flush 2024-05-08T15:58:16.064 INFO:tasks.workunit.client.0.smithi172.stdout:{} 2024-05-08T15:58:16.077 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:112: TEST_ec_profile_warning: grep 'WARNING: erasure coding profile prof-shec_generic uses plugin shec_generic' td/test-erasure-code-plugins/mon.a.log 2024-05-08T15:58:16.083 INFO:tasks.workunit.client.0.smithi172.stdout:2024-05-08T15:58:15.658+0000 7f360705c640 0 mon.a@0(leader).osd e26 WARNING: erasure coding profile prof-shec_generic uses plugin shec_generic that has been deprecated. Please use shec instead. 2024-05-08T15:58:16.083 INFO:tasks.workunit.client.0.smithi172.stdout:2024-05-08T15:58:15.950+0000 7f360705c640 0 mon.a@0(leader).osd e27 WARNING: erasure coding profile prof-shec_generic uses plugin shec_generic that has been deprecated. Please use shec instead. 2024-05-08T15:58:16.083 INFO:tasks.workunit.client.0.smithi172.stdout:2024-05-08T15:58:15.950+0000 7f360705c640 0 mon.a@0(leader).osd e27 WARNING: erasure coding profile prof-shec_generic uses plugin shec_generic that has been deprecated. Please use shec instead. 2024-05-08T15:58:16.084 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:109: TEST_ec_profile_warning: for plugin in ${legacy_shec_plugins[*]} 2024-05-08T15:58:16.084 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:110: TEST_ec_profile_warning: ceph osd erasure-code-profile set prof-shec_sse3 crush-failure-domain=osd plugin=shec_sse3 2024-05-08T15:58:16.622 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:111: TEST_ec_profile_warning: get_asok_path mon.a 2024-05-08T15:58:16.622 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-05-08T15:58:16.623 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-05-08T15:58:16.623 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-08T15:58:16.623 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T15:58:16.623 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20299 2024-05-08T15:58:16.624 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20299/ceph-mon.a.asok 2024-05-08T15:58:16.624 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:111: TEST_ec_profile_warning: CEPH_ARGS= 2024-05-08T15:58:16.624 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:111: TEST_ec_profile_warning: ceph --admin-daemon /tmp/ceph-asok.20299/ceph-mon.a.asok log flush 2024-05-08T15:58:16.713 INFO:tasks.workunit.client.0.smithi172.stdout:{} 2024-05-08T15:58:16.726 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:112: TEST_ec_profile_warning: grep 'WARNING: erasure coding profile prof-shec_sse3 uses plugin shec_sse3' td/test-erasure-code-plugins/mon.a.log 2024-05-08T15:58:16.731 INFO:tasks.workunit.client.0.smithi172.stdout:2024-05-08T15:58:16.374+0000 7f360705c640 0 mon.a@0(leader).osd e27 WARNING: erasure coding profile prof-shec_sse3 uses plugin shec_sse3 that has been deprecated. Please use shec instead. 2024-05-08T15:58:16.732 INFO:tasks.workunit.client.0.smithi172.stdout:2024-05-08T15:58:16.606+0000 7f360705c640 0 mon.a@0(leader).osd e28 WARNING: erasure coding profile prof-shec_sse3 uses plugin shec_sse3 that has been deprecated. Please use shec instead. 2024-05-08T15:58:16.732 INFO:tasks.workunit.client.0.smithi172.stdout:2024-05-08T15:58:16.606+0000 7f360705c640 0 mon.a@0(leader).osd e28 WARNING: erasure coding profile prof-shec_sse3 uses plugin shec_sse3 that has been deprecated. Please use shec instead. 2024-05-08T15:58:16.732 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:109: TEST_ec_profile_warning: for plugin in ${legacy_shec_plugins[*]} 2024-05-08T15:58:16.732 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:110: TEST_ec_profile_warning: ceph osd erasure-code-profile set prof-shec_sse4 crush-failure-domain=osd plugin=shec_sse4 2024-05-08T15:58:17.369 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:111: TEST_ec_profile_warning: get_asok_path mon.a 2024-05-08T15:58:17.369 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-05-08T15:58:17.370 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-05-08T15:58:17.370 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-08T15:58:17.371 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T15:58:17.371 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20299 2024-05-08T15:58:17.371 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20299/ceph-mon.a.asok 2024-05-08T15:58:17.371 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:111: TEST_ec_profile_warning: CEPH_ARGS= 2024-05-08T15:58:17.371 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:111: TEST_ec_profile_warning: ceph --admin-daemon /tmp/ceph-asok.20299/ceph-mon.a.asok log flush 2024-05-08T15:58:17.463 INFO:tasks.workunit.client.0.smithi172.stdout:{} 2024-05-08T15:58:17.475 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:112: TEST_ec_profile_warning: grep 'WARNING: erasure coding profile prof-shec_sse4 uses plugin shec_sse4' td/test-erasure-code-plugins/mon.a.log 2024-05-08T15:58:17.483 INFO:tasks.workunit.client.0.smithi172.stdout:2024-05-08T15:58:17.038+0000 7f360705c640 0 mon.a@0(leader).osd e28 WARNING: erasure coding profile prof-shec_sse4 uses plugin shec_sse4 that has been deprecated. Please use shec instead. 2024-05-08T15:58:17.484 INFO:tasks.workunit.client.0.smithi172.stdout:2024-05-08T15:58:17.354+0000 7f360705c640 0 mon.a@0(leader).osd e29 WARNING: erasure coding profile prof-shec_sse4 uses plugin shec_sse4 that has been deprecated. Please use shec instead. 2024-05-08T15:58:17.484 INFO:tasks.workunit.client.0.smithi172.stdout:2024-05-08T15:58:17.354+0000 7f360705c640 0 mon.a@0(leader).osd e29 WARNING: erasure coding profile prof-shec_sse4 uses plugin shec_sse4 that has been deprecated. Please use shec instead. 2024-05-08T15:58:17.486 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:115: TEST_ec_profile_warning: teardown td/test-erasure-code-plugins 2024-05-08T15:58:17.486 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-code-plugins 2024-05-08T15:58:17.486 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-05-08T15:58:17.486 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-code-plugins KILL 2024-05-08T15:58:17.487 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-08T15:58:17.487 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-08T15:58:17.487 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-08T15:58:17.487 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-08T15:58:17.487 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-08T15:58:17.594 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-08T15:58:17.595 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-05-08T15:58:17.596 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-05-08T15:58:17.597 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-05-08T15:58:17.597 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-05-08T15:58:17.598 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-05-08T15:58:17.598 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-05-08T15:58:17.599 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-08T15:58:17.599 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-05-08T15:58:17.599 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-05-08T15:58:17.599 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-08T15:58:17.600 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-05-08T15:58:17.601 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-05-08T15:58:17.601 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-code-plugins 2024-05-08T15:58:17.619 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-05-08T15:58:17.619 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T15:58:17.619 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20299 2024-05-08T15:58:17.619 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.20299 2024-05-08T15:58:17.621 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-05-08T15:58:17.621 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-05-08T15:58:17.621 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:35: run: for func in $funcs 2024-05-08T15:58:17.621 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:36: run: TEST_preload_no_warning td/test-erasure-code-plugins 2024-05-08T15:58:17.621 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:58: TEST_preload_no_warning: local dir=td/test-erasure-code-plugins 2024-05-08T15:58:17.621 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:60: TEST_preload_no_warning: for plugin in ${plugins[*]} 2024-05-08T15:58:17.621 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:61: TEST_preload_no_warning: setup td/test-erasure-code-plugins 2024-05-08T15:58:17.621 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/test-erasure-code-plugins 2024-05-08T15:58:17.621 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/test-erasure-code-plugins 2024-05-08T15:58:17.621 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-code-plugins 2024-05-08T15:58:17.621 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-05-08T15:58:17.621 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-code-plugins KILL 2024-05-08T15:58:17.622 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-08T15:58:17.622 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-08T15:58:17.622 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-08T15:58:17.622 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-08T15:58:17.622 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-08T15:58:17.622 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-08T15:58:17.623 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-05-08T15:58:17.623 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-05-08T15:58:17.623 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-05-08T15:58:17.625 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-05-08T15:58:17.625 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-05-08T15:58:17.625 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-05-08T15:58:17.626 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-08T15:58:17.626 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-05-08T15:58:17.626 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-05-08T15:58:17.626 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-08T15:58:17.627 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-05-08T15:58:17.628 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-05-08T15:58:17.628 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-code-plugins 2024-05-08T15:58:17.630 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-05-08T15:58:17.630 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T15:58:17.630 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20299 2024-05-08T15:58:17.630 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.20299 2024-05-08T15:58:17.631 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-05-08T15:58:17.631 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-05-08T15:58:17.631 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/test-erasure-code-plugins 2024-05-08T15:58:17.632 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-05-08T15:58:17.632 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T15:58:17.632 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20299 2024-05-08T15:58:17.632 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.20299 2024-05-08T15:58:17.634 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-05-08T15:58:17.634 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-05-08T15:58:17.634 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-05-08T15:58:17.634 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/test-erasure-code-plugins 1' TERM HUP INT 2024-05-08T15:58:17.634 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:62: TEST_preload_no_warning: run_mon td/test-erasure-code-plugins a --osd_erasure_code_plugins=jerasure 2024-05-08T15:58:17.634 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/test-erasure-code-plugins 2024-05-08T15:58:17.634 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-05-08T15:58:17.635 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-05-08T15:58:17.635 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-05-08T15:58:17.635 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/test-erasure-code-plugins/a 2024-05-08T15:58:17.635 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-code-plugins/a --run-dir=td/test-erasure-code-plugins --osd_erasure_code_plugins=jerasure 2024-05-08T15:58:18.063 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-05-08T15:58:18.063 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-08T15:58:18.063 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-08T15:58:18.063 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-08T15:58:18.063 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T15:58:18.063 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20299 2024-05-08T15:58:18.064 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.20299/$cluster-$name.asok' 2024-05-08T15:58:18.064 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-code-plugins/a '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.20299/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-code-plugins/log --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_erasure_code_plugins=jerasure 2024-05-08T15:58:18.273 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-05-08T15:58:18.275 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-05-08T15:58:18.275 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-05-08T15:58:18.275 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-05-08T15:58:18.275 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-05-08T15:58:18.276 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-05-08T15:58:18.276 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-05-08T15:58:18.276 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-05-08T15:58:18.276 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-05-08T15:58:18.277 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-08T15:58:18.277 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T15:58:18.277 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20299 2024-05-08T15:58:18.277 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20299/ceph-mon.a.asok 2024-05-08T15:58:18.277 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-05-08T15:58:18.277 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.20299/ceph-mon.a.asok config get fsid 2024-05-08T15:58:18.388 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-05-08T15:58:18.388 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-05-08T15:58:18.388 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-05-08T15:58:18.388 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-05-08T15:58:18.389 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-05-08T15:58:18.389 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-05-08T15:58:18.389 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-05-08T15:58:18.389 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-05-08T15:58:18.389 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-08T15:58:18.390 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T15:58:18.390 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20299 2024-05-08T15:58:18.390 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20299/ceph-mon.a.asok 2024-05-08T15:58:18.390 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-05-08T15:58:18.390 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.20299/ceph-mon.a.asok config get mon_host 2024-05-08T15:58:18.499 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:63: TEST_preload_no_warning: run_mgr td/test-erasure-code-plugins x 2024-05-08T15:58:18.499 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/test-erasure-code-plugins 2024-05-08T15:58:18.500 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-05-08T15:58:18.500 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-05-08T15:58:18.500 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-05-08T15:58:18.500 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/test-erasure-code-plugins/x 2024-05-08T15:58:18.500 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-05-08T15:58:19.401 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-05-08T15:58:19.402 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-08T15:58:19.402 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-08T15:58:19.402 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-08T15:58:19.402 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T15:58:19.403 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20299 2024-05-08T15:58:19.403 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.20299/$cluster-$name.asok' 2024-05-08T15:58:19.403 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-05-08T15:58:19.405 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-code-plugins/x '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.20299/$cluster-$name.asok' --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-05-08T15:58:19.437 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:64: TEST_preload_no_warning: get_asok_path mon.a 2024-05-08T15:58:19.437 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-05-08T15:58:19.437 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-05-08T15:58:19.437 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-08T15:58:19.437 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T15:58:19.437 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20299 2024-05-08T15:58:19.437 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20299/ceph-mon.a.asok 2024-05-08T15:58:19.438 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:64: TEST_preload_no_warning: CEPH_ARGS= 2024-05-08T15:58:19.438 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:64: TEST_preload_no_warning: ceph --admin-daemon /tmp/ceph-asok.20299/ceph-mon.a.asok log flush 2024-05-08T15:58:19.530 INFO:tasks.workunit.client.0.smithi172.stdout:{} 2024-05-08T15:58:19.543 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:65: TEST_preload_no_warning: run_osd td/test-erasure-code-plugins 0 --osd_erasure_code_plugins=jerasure 2024-05-08T15:58:19.543 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-code-plugins 2024-05-08T15:58:19.543 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-05-08T15:58:19.543 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-05-08T15:58:19.543 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-05-08T15:58:19.543 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-code-plugins/0 2024-05-08T15:58:19.543 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=2a3a20f9-46fd-4090-ba60-6d102331c696 --auth-supported=none --mon-host=127.0.0.1:17110 ' 2024-05-08T15:58:19.544 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-08T15:58:19.544 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-05-08T15:58:19.544 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-08T15:58:19.544 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-code-plugins/0' 2024-05-08T15:58:19.544 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code-plugins/0/journal' 2024-05-08T15:58:19.544 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-05-08T15:58:19.544 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-05-08T15:58:19.544 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-code-plugins' 2024-05-08T15:58:19.546 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-05-08T15:58:19.546 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-08T15:58:19.546 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-08T15:58:19.546 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-08T15:58:19.547 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T15:58:19.547 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20299 2024-05-08T15:58:19.547 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.20299/$cluster-$name.asok' 2024-05-08T15:58:19.547 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.20299/$cluster-$name.asok' 2024-05-08T15:58:19.547 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-05-08T15:58:19.547 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-05-08T15:58:19.547 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-05-08T15:58:19.547 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-code-plugins/$name.log' 2024-05-08T15:58:19.547 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-code-plugins/$name.pid' 2024-05-08T15:58:19.548 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-08T15:58:19.548 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-08T15:58:19.548 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-08T15:58:19.548 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-08T15:58:19.548 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-05-08T15:58:19.548 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_erasure_code_plugins=jerasure 2024-05-08T15:58:19.548 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-code-plugins/0 2024-05-08T15:58:19.549 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-05-08T15:58:19.549 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=981eafda-4d80-4e19-a761-0870e7147211 2024-05-08T15:58:19.549 INFO:tasks.workunit.client.0.smithi172.stdout:add osd0 981eafda-4d80-4e19-a761-0870e7147211 2024-05-08T15:58:19.549 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 981eafda-4d80-4e19-a761-0870e7147211' 2024-05-08T15:58:19.549 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-05-08T15:58:19.563 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAboTtm5tfdIRAAXeiPIPWD/7ZFwczDeFIMxw== 2024-05-08T15:58:19.563 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAboTtm5tfdIRAAXeiPIPWD/7ZFwczDeFIMxw=="}' 2024-05-08T15:58:19.564 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 981eafda-4d80-4e19-a761-0870e7147211 -i td/test-erasure-code-plugins/0/new.json 2024-05-08T15:58:19.835 INFO:tasks.workunit.client.0.smithi172.stdout:0 2024-05-08T15:58:19.850 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-code-plugins/0/new.json 2024-05-08T15:58:19.852 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=2a3a20f9-46fd-4090-ba60-6d102331c696 --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.20299/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_erasure_code_plugins=jerasure --mkfs --key AQAboTtm5tfdIRAAXeiPIPWD/7ZFwczDeFIMxw== --osd-uuid 981eafda-4d80-4e19-a761-0870e7147211 2024-05-08T15:58:19.887 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T15:58:19.890+0000 7f0d8c1317c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T15:58:19.903 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T15:58:19.906+0000 7f0d8c1317c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T15:58:19.904 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T15:58:19.906+0000 7f0d8c1317c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T15:58:19.905 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T15:58:19.906+0000 7f0d8c1317c0 -1 bluestore(td/test-erasure-code-plugins/0/block) _read_bdev_label failed to open td/test-erasure-code-plugins/0/block: (2) No such file or directory 2024-05-08T15:58:19.905 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T15:58:19.906+0000 7f0d8c1317c0 -1 bluestore(td/test-erasure-code-plugins/0/block) _read_bdev_label failed to open td/test-erasure-code-plugins/0/block: (2) No such file or directory 2024-05-08T15:58:19.905 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T15:58:19.906+0000 7f0d8c1317c0 -1 bluestore(td/test-erasure-code-plugins/0/block) _read_bdev_label failed to open td/test-erasure-code-plugins/0/block: (2) No such file or directory 2024-05-08T15:58:19.943 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T15:58:19.946+0000 7f0d8c1317c0 -1 bluestore(td/test-erasure-code-plugins/0) _read_fsid unparsable uuid 2024-05-08T15:58:26.211 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-code-plugins/0/keyring 2024-05-08T15:58:26.212 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-05-08T15:58:26.213 INFO:tasks.workunit.client.0.smithi172.stdout:adding osd0 key to auth repository 2024-05-08T15:58:26.213 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-05-08T15:58:26.213 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-code-plugins/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-05-08T15:58:26.762 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-05-08T15:58:26.763 INFO:tasks.workunit.client.0.smithi172.stdout:start osd.0 2024-05-08T15:58:26.763 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=2a3a20f9-46fd-4090-ba60-6d102331c696 --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.20299/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_erasure_code_plugins=jerasure 2024-05-08T15:58:26.763 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-05-08T15:58:26.764 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-05-08T15:58:26.764 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-05-08T15:58:26.792 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T15:58:26.794+0000 7f90485b67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T15:58:26.794 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T15:58:26.798+0000 7f90485b67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T15:58:26.795 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T15:58:26.798+0000 7f90485b67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T15:58:27.204 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-05-08T15:58:27.205 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-08T15:58:27.205 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-05-08T15:58:27.205 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-08T15:58:27.205 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-08T15:58:27.205 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T15:58:27.205 INFO:tasks.workunit.client.0.smithi172.stdout:0 2024-05-08T15:58:27.206 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-08T15:58:27.206 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T15:58:27.206 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T15:58:27.359 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T15:58:27.362+0000 7f90485b67c0 -1 Falling back to public interface 2024-05-08T15:58:27.648 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T15:58:28.650 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T15:58:28.650 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T15:58:28.651 INFO:tasks.workunit.client.0.smithi172.stdout:1 2024-05-08T15:58:28.651 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-08T15:58:28.651 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T15:58:28.651 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T15:58:29.106 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T15:58:29.991 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T15:58:29.994+0000 7f90485b67c0 -1 osd.0 0 log_to_monitors true 2024-05-08T15:58:30.108 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T15:58:30.108 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T15:58:30.109 INFO:tasks.workunit.client.0.smithi172.stdout:2 2024-05-08T15:58:30.109 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-08T15:58:30.109 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T15:58:30.109 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T15:58:30.716 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T15:58:31.717 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T15:58:31.718 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T15:58:31.718 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-05-08T15:58:31.718 INFO:tasks.workunit.client.0.smithi172.stdout:3 2024-05-08T15:58:31.718 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T15:58:31.718 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T15:58:32.122 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T15:58:33.123 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T15:58:33.124 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T15:58:33.124 INFO:tasks.workunit.client.0.smithi172.stdout:4 2024-05-08T15:58:33.124 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-05-08T15:58:33.124 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T15:58:33.125 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T15:58:33.578 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T15:58:34.579 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T15:58:34.579 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T15:58:34.579 INFO:tasks.workunit.client.0.smithi172.stdout:5 2024-05-08T15:58:34.580 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-05-08T15:58:34.580 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T15:58:34.580 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T15:58:35.159 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T15:58:36.161 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T15:58:36.161 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T15:58:36.161 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-05-08T15:58:36.161 INFO:tasks.workunit.client.0.smithi172.stdout:6 2024-05-08T15:58:36.162 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T15:58:36.162 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T15:58:36.579 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T15:58:37.581 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T15:58:37.581 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T15:58:37.582 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-05-08T15:58:37.582 INFO:tasks.workunit.client.0.smithi172.stdout:7 2024-05-08T15:58:37.582 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T15:58:37.582 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T15:58:38.026 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T15:58:39.027 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T15:58:39.027 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T15:58:39.028 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-05-08T15:58:39.028 INFO:tasks.workunit.client.0.smithi172.stdout:8 2024-05-08T15:58:39.028 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T15:58:39.028 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T15:58:39.484 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T15:58:40.486 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T15:58:40.487 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T15:58:40.487 INFO:tasks.workunit.client.0.smithi172.stdout:9 2024-05-08T15:58:40.487 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-05-08T15:58:40.487 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T15:58:40.487 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T15:58:40.911 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T15:58:41.913 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T15:58:41.913 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T15:58:41.913 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-05-08T15:58:41.913 INFO:tasks.workunit.client.0.smithi172.stdout:10 2024-05-08T15:58:41.914 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T15:58:41.914 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T15:58:42.368 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T15:58:43.370 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T15:58:43.370 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T15:58:43.370 INFO:tasks.workunit.client.0.smithi172.stdout:11 2024-05-08T15:58:43.371 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-05-08T15:58:43.371 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T15:58:43.371 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T15:58:44.468 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T15:58:45.469 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T15:58:45.469 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T15:58:45.470 INFO:tasks.workunit.client.0.smithi172.stdout:12 2024-05-08T15:58:45.470 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-05-08T15:58:45.470 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T15:58:45.470 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T15:58:46.537 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T15:58:47.538 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T15:58:47.539 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T15:58:47.539 INFO:tasks.workunit.client.0.smithi172.stdout:13 2024-05-08T15:58:47.539 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-05-08T15:58:47.540 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T15:58:47.540 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T15:58:48.278 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T15:58:49.280 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T15:58:49.280 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T15:58:49.281 INFO:tasks.workunit.client.0.smithi172.stdout:14 2024-05-08T15:58:49.281 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-05-08T15:58:49.281 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T15:58:49.281 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T15:58:50.137 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T15:58:51.139 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T15:58:51.139 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T15:58:51.139 INFO:tasks.workunit.client.0.smithi172.stdout:15 2024-05-08T15:58:51.140 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-05-08T15:58:51.140 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T15:58:51.140 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T15:58:51.944 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T15:58:52.946 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T15:58:52.946 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T15:58:52.947 INFO:tasks.workunit.client.0.smithi172.stdout:16 2024-05-08T15:58:52.947 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-05-08T15:58:52.947 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T15:58:52.947 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T15:58:53.606 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T15:58:54.607 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T15:58:54.608 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T15:58:54.608 INFO:tasks.workunit.client.0.smithi172.stdout:17 2024-05-08T15:58:54.608 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-05-08T15:58:54.608 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T15:58:54.608 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T15:58:55.052 INFO:tasks.workunit.client.0.smithi172.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2061380502,v1:127.0.0.1:6803/2061380502] [v2:127.0.0.1:6804/2061380502,v1:127.0.0.1:6805/2061380502] exists,up 981eafda-4d80-4e19-a761-0870e7147211 2024-05-08T15:58:55.054 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-08T15:58:55.054 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-08T15:58:55.054 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-08T15:58:55.054 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:66: TEST_preload_no_warning: get_asok_path osd.0 2024-05-08T15:58:55.054 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-05-08T15:58:55.054 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-05-08T15:58:55.054 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-08T15:58:55.055 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T15:58:55.055 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20299 2024-05-08T15:58:55.055 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20299/ceph-osd.0.asok 2024-05-08T15:58:55.055 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:66: TEST_preload_no_warning: CEPH_ARGS= 2024-05-08T15:58:55.055 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:66: TEST_preload_no_warning: ceph --admin-daemon /tmp/ceph-asok.20299/ceph-osd.0.asok log flush 2024-05-08T15:58:55.164 INFO:tasks.workunit.client.0.smithi172.stdout:{} 2024-05-08T15:58:55.177 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:67: TEST_preload_no_warning: grep 'WARNING: osd_erasure_code_plugins contains plugin' td/test-erasure-code-plugins/mon.a.log 2024-05-08T15:58:55.181 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:68: TEST_preload_no_warning: grep 'WARNING: osd_erasure_code_plugins contains plugin' td/test-erasure-code-plugins/osd.0.log 2024-05-08T15:58:55.183 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:69: TEST_preload_no_warning: teardown td/test-erasure-code-plugins 2024-05-08T15:58:55.183 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-code-plugins 2024-05-08T15:58:55.183 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-05-08T15:58:55.183 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-code-plugins KILL 2024-05-08T15:58:55.184 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-08T15:58:55.184 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-08T15:58:55.184 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-08T15:58:55.184 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-08T15:58:55.184 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-08T15:58:55.493 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-08T15:58:55.494 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-05-08T15:58:55.494 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-05-08T15:58:55.495 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-05-08T15:58:55.496 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-05-08T15:58:55.496 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-05-08T15:58:55.496 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-05-08T15:58:55.497 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-08T15:58:55.497 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-05-08T15:58:55.498 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-05-08T15:58:55.498 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-08T15:58:55.499 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-05-08T15:58:55.500 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-05-08T15:58:55.500 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-code-plugins 2024-05-08T15:58:55.508 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-05-08T15:58:55.508 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T15:58:55.508 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20299 2024-05-08T15:58:55.508 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.20299 2024-05-08T15:58:55.510 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-05-08T15:58:55.510 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-05-08T15:58:55.511 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:60: TEST_preload_no_warning: for plugin in ${plugins[*]} 2024-05-08T15:58:55.511 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:61: TEST_preload_no_warning: setup td/test-erasure-code-plugins 2024-05-08T15:58:55.511 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/test-erasure-code-plugins 2024-05-08T15:58:55.511 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/test-erasure-code-plugins 2024-05-08T15:58:55.511 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-code-plugins 2024-05-08T15:58:55.511 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-05-08T15:58:55.511 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-code-plugins KILL 2024-05-08T15:58:55.511 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-08T15:58:55.511 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-08T15:58:55.511 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-08T15:58:55.511 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-08T15:58:55.511 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-08T15:58:55.514 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-08T15:58:55.514 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-05-08T15:58:55.515 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-05-08T15:58:55.515 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-05-08T15:58:55.517 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-05-08T15:58:55.517 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-05-08T15:58:55.517 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-05-08T15:58:55.518 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-08T15:58:55.518 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-05-08T15:58:55.519 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-05-08T15:58:55.519 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-08T15:58:55.520 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-05-08T15:58:55.521 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-05-08T15:58:55.521 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-code-plugins 2024-05-08T15:58:55.522 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-05-08T15:58:55.522 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T15:58:55.523 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20299 2024-05-08T15:58:55.523 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.20299 2024-05-08T15:58:55.524 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-05-08T15:58:55.524 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-05-08T15:58:55.524 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/test-erasure-code-plugins 2024-05-08T15:58:55.525 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-05-08T15:58:55.525 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T15:58:55.526 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20299 2024-05-08T15:58:55.526 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.20299 2024-05-08T15:58:55.527 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-05-08T15:58:55.528 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-05-08T15:58:55.528 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-05-08T15:58:55.528 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/test-erasure-code-plugins 1' TERM HUP INT 2024-05-08T15:58:55.528 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:62: TEST_preload_no_warning: run_mon td/test-erasure-code-plugins a --osd_erasure_code_plugins=shec 2024-05-08T15:58:55.528 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/test-erasure-code-plugins 2024-05-08T15:58:55.529 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-05-08T15:58:55.529 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-05-08T15:58:55.529 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-05-08T15:58:55.529 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/test-erasure-code-plugins/a 2024-05-08T15:58:55.529 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-code-plugins/a --run-dir=td/test-erasure-code-plugins --osd_erasure_code_plugins=shec 2024-05-08T15:58:55.927 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-05-08T15:58:55.927 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-08T15:58:55.927 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-08T15:58:55.928 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-08T15:58:55.928 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T15:58:55.928 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20299 2024-05-08T15:58:55.928 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.20299/$cluster-$name.asok' 2024-05-08T15:58:55.928 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-code-plugins/a '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.20299/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-code-plugins/log --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_erasure_code_plugins=shec 2024-05-08T15:58:56.146 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-05-08T15:58:56.147 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-05-08T15:58:56.147 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-05-08T15:58:56.147 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-05-08T15:58:56.147 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-05-08T15:58:56.147 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-05-08T15:58:56.148 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-05-08T15:58:56.148 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-05-08T15:58:56.148 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-05-08T15:58:56.149 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-08T15:58:56.149 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T15:58:56.149 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20299 2024-05-08T15:58:56.149 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20299/ceph-mon.a.asok 2024-05-08T15:58:56.149 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-05-08T15:58:56.149 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.20299/ceph-mon.a.asok config get fsid 2024-05-08T15:58:56.263 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-05-08T15:58:56.263 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-05-08T15:58:56.263 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-05-08T15:58:56.263 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-05-08T15:58:56.264 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-05-08T15:58:56.264 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-05-08T15:58:56.264 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-05-08T15:58:56.264 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-05-08T15:58:56.265 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-08T15:58:56.265 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T15:58:56.265 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20299 2024-05-08T15:58:56.265 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20299/ceph-mon.a.asok 2024-05-08T15:58:56.265 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-05-08T15:58:56.265 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.20299/ceph-mon.a.asok config get mon_host 2024-05-08T15:58:56.366 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:63: TEST_preload_no_warning: run_mgr td/test-erasure-code-plugins x 2024-05-08T15:58:56.367 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/test-erasure-code-plugins 2024-05-08T15:58:56.367 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-05-08T15:58:56.367 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-05-08T15:58:56.367 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-05-08T15:58:56.367 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/test-erasure-code-plugins/x 2024-05-08T15:58:56.367 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-05-08T15:58:57.267 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-05-08T15:58:57.267 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-08T15:58:57.267 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-08T15:58:57.268 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-08T15:58:57.268 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T15:58:57.268 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20299 2024-05-08T15:58:57.268 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.20299/$cluster-$name.asok' 2024-05-08T15:58:57.269 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-05-08T15:58:57.269 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-code-plugins/x '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.20299/$cluster-$name.asok' --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-05-08T15:58:57.302 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:64: TEST_preload_no_warning: get_asok_path mon.a 2024-05-08T15:58:57.302 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-05-08T15:58:57.302 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-05-08T15:58:57.302 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-08T15:58:57.303 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T15:58:57.303 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20299 2024-05-08T15:58:57.303 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20299/ceph-mon.a.asok 2024-05-08T15:58:57.303 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:64: TEST_preload_no_warning: CEPH_ARGS= 2024-05-08T15:58:57.303 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:64: TEST_preload_no_warning: ceph --admin-daemon /tmp/ceph-asok.20299/ceph-mon.a.asok log flush 2024-05-08T15:58:57.400 INFO:tasks.workunit.client.0.smithi172.stdout:{} 2024-05-08T15:58:57.414 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:65: TEST_preload_no_warning: run_osd td/test-erasure-code-plugins 0 --osd_erasure_code_plugins=shec 2024-05-08T15:58:57.414 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-code-plugins 2024-05-08T15:58:57.414 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-05-08T15:58:57.415 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-05-08T15:58:57.415 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-05-08T15:58:57.415 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-code-plugins/0 2024-05-08T15:58:57.415 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=2a3a20f9-46fd-4090-ba60-6d102331c696 --auth-supported=none --mon-host=127.0.0.1:17110 ' 2024-05-08T15:58:57.415 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-08T15:58:57.415 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-05-08T15:58:57.415 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-08T15:58:57.415 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-code-plugins/0' 2024-05-08T15:58:57.415 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code-plugins/0/journal' 2024-05-08T15:58:57.415 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-05-08T15:58:57.415 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-05-08T15:58:57.415 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-code-plugins' 2024-05-08T15:58:57.415 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-05-08T15:58:57.415 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-08T15:58:57.415 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-08T15:58:57.416 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-08T15:58:57.416 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T15:58:57.416 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20299 2024-05-08T15:58:57.416 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.20299/$cluster-$name.asok' 2024-05-08T15:58:57.418 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.20299/$cluster-$name.asok' 2024-05-08T15:58:57.418 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-05-08T15:58:57.418 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-05-08T15:58:57.418 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-05-08T15:58:57.418 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-code-plugins/$name.log' 2024-05-08T15:58:57.418 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-code-plugins/$name.pid' 2024-05-08T15:58:57.418 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-08T15:58:57.418 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-08T15:58:57.418 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-08T15:58:57.418 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-08T15:58:57.419 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-05-08T15:58:57.419 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_erasure_code_plugins=shec 2024-05-08T15:58:57.419 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-code-plugins/0 2024-05-08T15:58:57.419 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-05-08T15:58:57.419 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=ac446352-bd30-4cae-812f-cd59b20fe7ff 2024-05-08T15:58:57.419 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 ac446352-bd30-4cae-812f-cd59b20fe7ff' 2024-05-08T15:58:57.419 INFO:tasks.workunit.client.0.smithi172.stdout:add osd0 ac446352-bd30-4cae-812f-cd59b20fe7ff 2024-05-08T15:58:57.419 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-05-08T15:58:57.434 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBBoTtmc44lGhAAGD0Q8aM4l9qzFjGA0GItBw== 2024-05-08T15:58:57.434 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBBoTtmc44lGhAAGD0Q8aM4l9qzFjGA0GItBw=="}' 2024-05-08T15:58:57.434 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new ac446352-bd30-4cae-812f-cd59b20fe7ff -i td/test-erasure-code-plugins/0/new.json 2024-05-08T15:58:57.715 INFO:tasks.workunit.client.0.smithi172.stdout:0 2024-05-08T15:58:57.730 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-code-plugins/0/new.json 2024-05-08T15:58:57.731 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=2a3a20f9-46fd-4090-ba60-6d102331c696 --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.20299/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_erasure_code_plugins=shec --mkfs --key AQBBoTtmc44lGhAAGD0Q8aM4l9qzFjGA0GItBw== --osd-uuid ac446352-bd30-4cae-812f-cd59b20fe7ff 2024-05-08T15:58:57.763 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T15:58:57.765+0000 7ff84ac667c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T15:58:57.785 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T15:58:57.785+0000 7ff84ac667c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T15:58:57.786 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T15:58:57.789+0000 7ff84ac667c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T15:58:57.787 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T15:58:57.789+0000 7ff84ac667c0 -1 bluestore(td/test-erasure-code-plugins/0/block) _read_bdev_label failed to open td/test-erasure-code-plugins/0/block: (2) No such file or directory 2024-05-08T15:58:57.787 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T15:58:57.789+0000 7ff84ac667c0 -1 bluestore(td/test-erasure-code-plugins/0/block) _read_bdev_label failed to open td/test-erasure-code-plugins/0/block: (2) No such file or directory 2024-05-08T15:58:57.787 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T15:58:57.789+0000 7ff84ac667c0 -1 bluestore(td/test-erasure-code-plugins/0/block) _read_bdev_label failed to open td/test-erasure-code-plugins/0/block: (2) No such file or directory 2024-05-08T15:58:57.815 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T15:58:57.817+0000 7ff84ac667c0 -1 bluestore(td/test-erasure-code-plugins/0) _read_fsid unparsable uuid 2024-05-08T15:59:04.048 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-code-plugins/0/keyring 2024-05-08T15:59:04.048 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-05-08T15:59:04.049 INFO:tasks.workunit.client.0.smithi172.stdout:adding osd0 key to auth repository 2024-05-08T15:59:04.049 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-05-08T15:59:04.050 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-code-plugins/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-05-08T15:59:04.591 INFO:tasks.workunit.client.0.smithi172.stdout:start osd.0 2024-05-08T15:59:04.592 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-05-08T15:59:04.592 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-05-08T15:59:04.592 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-05-08T15:59:04.592 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=2a3a20f9-46fd-4090-ba60-6d102331c696 --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.20299/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_erasure_code_plugins=shec 2024-05-08T15:59:04.592 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-05-08T15:59:04.622 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T15:59:04.625+0000 7f7b787c97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T15:59:04.624 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T15:59:04.625+0000 7f7b787c97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T15:59:04.625 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T15:59:04.625+0000 7f7b787c97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T15:59:05.061 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-05-08T15:59:05.061 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-08T15:59:05.061 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-05-08T15:59:05.061 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-08T15:59:05.061 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-08T15:59:05.061 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T15:59:05.062 INFO:tasks.workunit.client.0.smithi172.stdout:0 2024-05-08T15:59:05.062 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-08T15:59:05.062 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T15:59:05.062 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T15:59:05.187 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T15:59:05.189+0000 7f7b787c97c0 -1 Falling back to public interface 2024-05-08T15:59:05.475 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T15:59:06.476 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T15:59:06.477 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T15:59:06.477 INFO:tasks.workunit.client.0.smithi172.stdout:1 2024-05-08T15:59:06.477 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-08T15:59:06.478 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T15:59:06.478 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T15:59:06.876 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T15:59:07.814 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T15:59:07.813+0000 7f7b787c97c0 -1 osd.0 0 log_to_monitors true 2024-05-08T15:59:07.877 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T15:59:07.877 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T15:59:07.877 INFO:tasks.workunit.client.0.smithi172.stdout:2 2024-05-08T15:59:07.878 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-08T15:59:07.878 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T15:59:07.878 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T15:59:08.531 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T15:59:09.534 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T15:59:09.534 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T15:59:09.535 INFO:tasks.workunit.client.0.smithi172.stdout:3 2024-05-08T15:59:09.536 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-05-08T15:59:09.536 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T15:59:09.536 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T15:59:09.986 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T15:59:10.988 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T15:59:10.988 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T15:59:10.988 INFO:tasks.workunit.client.0.smithi172.stdout:4 2024-05-08T15:59:10.989 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-05-08T15:59:10.989 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T15:59:10.989 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T15:59:11.433 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T15:59:12.434 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T15:59:12.435 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T15:59:12.435 INFO:tasks.workunit.client.0.smithi172.stdout:5 2024-05-08T15:59:12.435 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-05-08T15:59:12.436 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T15:59:12.436 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T15:59:13.022 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T15:59:14.024 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T15:59:14.024 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T15:59:14.024 INFO:tasks.workunit.client.0.smithi172.stdout:6 2024-05-08T15:59:14.025 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-05-08T15:59:14.025 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T15:59:14.025 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T15:59:14.450 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T15:59:15.451 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T15:59:15.452 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T15:59:15.452 INFO:tasks.workunit.client.0.smithi172.stdout:7 2024-05-08T15:59:15.452 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-05-08T15:59:15.452 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T15:59:15.452 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T15:59:15.996 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T15:59:16.997 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T15:59:16.997 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T15:59:16.998 INFO:tasks.workunit.client.0.smithi172.stdout:8 2024-05-08T15:59:16.998 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-05-08T15:59:16.998 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T15:59:16.998 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T15:59:17.462 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T15:59:18.463 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T15:59:18.463 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T15:59:18.464 INFO:tasks.workunit.client.0.smithi172.stdout:9 2024-05-08T15:59:18.464 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-05-08T15:59:18.464 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T15:59:18.464 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T15:59:18.976 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T15:59:19.978 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T15:59:19.978 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T15:59:19.978 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-05-08T15:59:19.979 INFO:tasks.workunit.client.0.smithi172.stdout:10 2024-05-08T15:59:19.979 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T15:59:19.980 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T15:59:20.425 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T15:59:21.427 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T15:59:21.427 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T15:59:21.427 INFO:tasks.workunit.client.0.smithi172.stdout:11 2024-05-08T15:59:21.428 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-05-08T15:59:21.428 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T15:59:21.428 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T15:59:22.206 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T15:59:23.208 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T15:59:23.208 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T15:59:23.208 INFO:tasks.workunit.client.0.smithi172.stdout:12 2024-05-08T15:59:23.209 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-05-08T15:59:23.209 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T15:59:23.209 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T15:59:24.040 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T15:59:25.042 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T15:59:25.043 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T15:59:25.043 INFO:tasks.workunit.client.0.smithi172.stdout:13 2024-05-08T15:59:25.043 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-05-08T15:59:25.043 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T15:59:25.044 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T15:59:25.857 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T15:59:26.859 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T15:59:26.859 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T15:59:26.859 INFO:tasks.workunit.client.0.smithi172.stdout:14 2024-05-08T15:59:26.860 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-05-08T15:59:26.860 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T15:59:26.860 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T15:59:27.842 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T15:59:28.843 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T15:59:28.844 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T15:59:28.844 INFO:tasks.workunit.client.0.smithi172.stdout:15 2024-05-08T15:59:28.844 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-05-08T15:59:28.845 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T15:59:28.845 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T15:59:29.935 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T15:59:30.936 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T15:59:30.936 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T15:59:30.937 INFO:tasks.workunit.client.0.smithi172.stdout:16 2024-05-08T15:59:30.937 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-05-08T15:59:30.937 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T15:59:30.937 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T15:59:31.846 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T15:59:32.848 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T15:59:32.849 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T15:59:32.849 INFO:tasks.workunit.client.0.smithi172.stdout:17 2024-05-08T15:59:32.849 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-05-08T15:59:32.849 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T15:59:32.849 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T15:59:33.257 INFO:tasks.workunit.client.0.smithi172.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/157887914,v1:127.0.0.1:6803/157887914] [v2:127.0.0.1:6804/157887914,v1:127.0.0.1:6805/157887914] exists,up ac446352-bd30-4cae-812f-cd59b20fe7ff 2024-05-08T15:59:33.259 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-08T15:59:33.259 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-08T15:59:33.259 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-08T15:59:33.259 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:66: TEST_preload_no_warning: get_asok_path osd.0 2024-05-08T15:59:33.259 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-05-08T15:59:33.259 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-05-08T15:59:33.259 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-08T15:59:33.259 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T15:59:33.259 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20299 2024-05-08T15:59:33.259 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20299/ceph-osd.0.asok 2024-05-08T15:59:33.259 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:66: TEST_preload_no_warning: CEPH_ARGS= 2024-05-08T15:59:33.260 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:66: TEST_preload_no_warning: ceph --admin-daemon /tmp/ceph-asok.20299/ceph-osd.0.asok log flush 2024-05-08T15:59:33.369 INFO:tasks.workunit.client.0.smithi172.stdout:{} 2024-05-08T15:59:33.382 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:67: TEST_preload_no_warning: grep 'WARNING: osd_erasure_code_plugins contains plugin' td/test-erasure-code-plugins/mon.a.log 2024-05-08T15:59:33.387 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:68: TEST_preload_no_warning: grep 'WARNING: osd_erasure_code_plugins contains plugin' td/test-erasure-code-plugins/osd.0.log 2024-05-08T15:59:33.390 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:69: TEST_preload_no_warning: teardown td/test-erasure-code-plugins 2024-05-08T15:59:33.390 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-code-plugins 2024-05-08T15:59:33.390 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-05-08T15:59:33.390 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-code-plugins KILL 2024-05-08T15:59:33.391 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-08T15:59:33.391 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-08T15:59:33.391 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-08T15:59:33.391 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-08T15:59:33.391 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-08T15:59:33.499 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-08T15:59:33.500 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-05-08T15:59:33.500 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-05-08T15:59:33.501 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-05-08T15:59:33.502 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-05-08T15:59:33.502 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-05-08T15:59:33.503 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-05-08T15:59:33.504 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-08T15:59:33.504 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-05-08T15:59:33.504 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-05-08T15:59:33.504 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-08T15:59:33.505 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-05-08T15:59:33.506 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-05-08T15:59:33.506 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-code-plugins 2024-05-08T15:59:33.512 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-05-08T15:59:33.512 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T15:59:33.512 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20299 2024-05-08T15:59:33.513 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.20299 2024-05-08T15:59:33.514 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-05-08T15:59:33.514 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-05-08T15:59:33.515 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:60: TEST_preload_no_warning: for plugin in ${plugins[*]} 2024-05-08T15:59:33.515 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:61: TEST_preload_no_warning: setup td/test-erasure-code-plugins 2024-05-08T15:59:33.515 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/test-erasure-code-plugins 2024-05-08T15:59:33.515 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/test-erasure-code-plugins 2024-05-08T15:59:33.515 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-code-plugins 2024-05-08T15:59:33.515 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-05-08T15:59:33.515 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-code-plugins KILL 2024-05-08T15:59:33.515 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-08T15:59:33.515 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-08T15:59:33.515 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-08T15:59:33.515 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-08T15:59:33.515 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-08T15:59:33.516 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-08T15:59:33.516 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-05-08T15:59:33.517 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-05-08T15:59:33.517 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-05-08T15:59:33.518 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-05-08T15:59:33.518 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-05-08T15:59:33.519 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-05-08T15:59:33.519 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-08T15:59:33.519 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-05-08T15:59:33.520 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-05-08T15:59:33.520 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-08T15:59:33.521 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-05-08T15:59:33.522 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-05-08T15:59:33.522 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-code-plugins 2024-05-08T15:59:33.523 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-05-08T15:59:33.523 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T15:59:33.523 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20299 2024-05-08T15:59:33.523 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.20299 2024-05-08T15:59:33.524 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-05-08T15:59:33.524 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-05-08T15:59:33.525 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/test-erasure-code-plugins 2024-05-08T15:59:33.525 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-05-08T15:59:33.525 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T15:59:33.525 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20299 2024-05-08T15:59:33.526 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.20299 2024-05-08T15:59:33.527 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-05-08T15:59:33.527 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-05-08T15:59:33.528 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-05-08T15:59:33.528 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/test-erasure-code-plugins 1' TERM HUP INT 2024-05-08T15:59:33.528 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:62: TEST_preload_no_warning: run_mon td/test-erasure-code-plugins a --osd_erasure_code_plugins=lrc 2024-05-08T15:59:33.528 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/test-erasure-code-plugins 2024-05-08T15:59:33.528 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-05-08T15:59:33.528 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-05-08T15:59:33.528 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-05-08T15:59:33.528 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/test-erasure-code-plugins/a 2024-05-08T15:59:33.528 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-code-plugins/a --run-dir=td/test-erasure-code-plugins --osd_erasure_code_plugins=lrc 2024-05-08T15:59:33.926 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-05-08T15:59:33.926 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-08T15:59:33.926 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-08T15:59:33.927 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-08T15:59:33.927 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T15:59:33.927 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20299 2024-05-08T15:59:33.927 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.20299/$cluster-$name.asok' 2024-05-08T15:59:33.928 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-code-plugins/a '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.20299/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-code-plugins/log --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_erasure_code_plugins=lrc 2024-05-08T15:59:34.155 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-05-08T15:59:34.156 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-05-08T15:59:34.156 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-05-08T15:59:34.156 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-05-08T15:59:34.156 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-05-08T15:59:34.157 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-05-08T15:59:34.157 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-05-08T15:59:34.157 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-05-08T15:59:34.157 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-05-08T15:59:34.157 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-08T15:59:34.158 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T15:59:34.158 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20299 2024-05-08T15:59:34.158 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20299/ceph-mon.a.asok 2024-05-08T15:59:34.158 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-05-08T15:59:34.158 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.20299/ceph-mon.a.asok config get fsid 2024-05-08T15:59:34.274 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-05-08T15:59:34.274 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-05-08T15:59:34.274 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-05-08T15:59:34.274 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-05-08T15:59:34.274 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-05-08T15:59:34.275 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-05-08T15:59:34.275 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-05-08T15:59:34.275 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-05-08T15:59:34.275 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-08T15:59:34.276 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T15:59:34.276 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20299 2024-05-08T15:59:34.276 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20299/ceph-mon.a.asok 2024-05-08T15:59:34.276 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-05-08T15:59:34.276 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.20299/ceph-mon.a.asok config get mon_host 2024-05-08T15:59:34.385 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:63: TEST_preload_no_warning: run_mgr td/test-erasure-code-plugins x 2024-05-08T15:59:34.386 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/test-erasure-code-plugins 2024-05-08T15:59:34.386 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-05-08T15:59:34.386 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-05-08T15:59:34.386 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-05-08T15:59:34.386 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/test-erasure-code-plugins/x 2024-05-08T15:59:34.386 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-05-08T15:59:35.285 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-05-08T15:59:35.285 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-08T15:59:35.285 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-08T15:59:35.285 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-08T15:59:35.285 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T15:59:35.285 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20299 2024-05-08T15:59:35.285 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.20299/$cluster-$name.asok' 2024-05-08T15:59:35.286 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-05-08T15:59:35.286 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-code-plugins/x '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.20299/$cluster-$name.asok' --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-05-08T15:59:35.320 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:64: TEST_preload_no_warning: get_asok_path mon.a 2024-05-08T15:59:35.320 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-05-08T15:59:35.320 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-05-08T15:59:35.320 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-08T15:59:35.320 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T15:59:35.320 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20299 2024-05-08T15:59:35.320 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20299/ceph-mon.a.asok 2024-05-08T15:59:35.320 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:64: TEST_preload_no_warning: CEPH_ARGS= 2024-05-08T15:59:35.320 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:64: TEST_preload_no_warning: ceph --admin-daemon /tmp/ceph-asok.20299/ceph-mon.a.asok log flush 2024-05-08T15:59:35.418 INFO:tasks.workunit.client.0.smithi172.stdout:{} 2024-05-08T15:59:35.434 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:65: TEST_preload_no_warning: run_osd td/test-erasure-code-plugins 0 --osd_erasure_code_plugins=lrc 2024-05-08T15:59:35.434 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-code-plugins 2024-05-08T15:59:35.434 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-05-08T15:59:35.434 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-05-08T15:59:35.434 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-05-08T15:59:35.434 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-code-plugins/0 2024-05-08T15:59:35.434 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=2a3a20f9-46fd-4090-ba60-6d102331c696 --auth-supported=none --mon-host=127.0.0.1:17110 ' 2024-05-08T15:59:35.434 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-08T15:59:35.434 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-05-08T15:59:35.434 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-08T15:59:35.434 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-code-plugins/0' 2024-05-08T15:59:35.435 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code-plugins/0/journal' 2024-05-08T15:59:35.435 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-05-08T15:59:35.435 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-05-08T15:59:35.435 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-code-plugins' 2024-05-08T15:59:35.435 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-05-08T15:59:35.435 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-08T15:59:35.435 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-08T15:59:35.437 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-08T15:59:35.437 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T15:59:35.437 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20299 2024-05-08T15:59:35.437 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.20299/$cluster-$name.asok' 2024-05-08T15:59:35.437 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.20299/$cluster-$name.asok' 2024-05-08T15:59:35.438 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-05-08T15:59:35.438 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-05-08T15:59:35.438 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-05-08T15:59:35.438 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-code-plugins/$name.log' 2024-05-08T15:59:35.438 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-code-plugins/$name.pid' 2024-05-08T15:59:35.438 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-08T15:59:35.438 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-08T15:59:35.438 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-08T15:59:35.438 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-08T15:59:35.438 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-05-08T15:59:35.438 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_erasure_code_plugins=lrc 2024-05-08T15:59:35.438 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-code-plugins/0 2024-05-08T15:59:35.438 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-05-08T15:59:35.439 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=1db2c695-418f-426b-ae35-c3b454f418d3 2024-05-08T15:59:35.439 INFO:tasks.workunit.client.0.smithi172.stdout:add osd0 1db2c695-418f-426b-ae35-c3b454f418d3 2024-05-08T15:59:35.440 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 1db2c695-418f-426b-ae35-c3b454f418d3' 2024-05-08T15:59:35.440 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-05-08T15:59:35.459 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBnoTtmwLSkGxAA2WI706nKHmWfWGBA4hv1Mw== 2024-05-08T15:59:35.459 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBnoTtmwLSkGxAA2WI706nKHmWfWGBA4hv1Mw=="}' 2024-05-08T15:59:35.459 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 1db2c695-418f-426b-ae35-c3b454f418d3 -i td/test-erasure-code-plugins/0/new.json 2024-05-08T15:59:35.757 INFO:tasks.workunit.client.0.smithi172.stdout:0 2024-05-08T15:59:35.773 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-code-plugins/0/new.json 2024-05-08T15:59:35.773 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=2a3a20f9-46fd-4090-ba60-6d102331c696 --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.20299/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_erasure_code_plugins=lrc --mkfs --key AQBnoTtmwLSkGxAA2WI706nKHmWfWGBA4hv1Mw== --osd-uuid 1db2c695-418f-426b-ae35-c3b454f418d3 2024-05-08T15:59:35.800 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T15:59:35.801+0000 7f4e2b5577c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T15:59:35.826 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T15:59:35.829+0000 7f4e2b5577c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T15:59:35.827 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T15:59:35.829+0000 7f4e2b5577c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T15:59:35.828 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T15:59:35.829+0000 7f4e2b5577c0 -1 bluestore(td/test-erasure-code-plugins/0/block) _read_bdev_label failed to open td/test-erasure-code-plugins/0/block: (2) No such file or directory 2024-05-08T15:59:35.828 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T15:59:35.829+0000 7f4e2b5577c0 -1 bluestore(td/test-erasure-code-plugins/0/block) _read_bdev_label failed to open td/test-erasure-code-plugins/0/block: (2) No such file or directory 2024-05-08T15:59:35.828 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T15:59:35.829+0000 7f4e2b5577c0 -1 bluestore(td/test-erasure-code-plugins/0/block) _read_bdev_label failed to open td/test-erasure-code-plugins/0/block: (2) No such file or directory 2024-05-08T15:59:35.866 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T15:59:35.869+0000 7f4e2b5577c0 -1 bluestore(td/test-erasure-code-plugins/0) _read_fsid unparsable uuid 2024-05-08T15:59:42.120 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-code-plugins/0/keyring 2024-05-08T15:59:42.121 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-05-08T15:59:42.122 INFO:tasks.workunit.client.0.smithi172.stdout:adding osd0 key to auth repository 2024-05-08T15:59:42.122 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-05-08T15:59:42.123 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-code-plugins/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-05-08T15:59:42.658 INFO:tasks.workunit.client.0.smithi172.stdout:start osd.0 2024-05-08T15:59:42.660 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-05-08T15:59:42.660 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=2a3a20f9-46fd-4090-ba60-6d102331c696 --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.20299/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_erasure_code_plugins=lrc 2024-05-08T15:59:42.660 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-05-08T15:59:42.660 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-05-08T15:59:42.660 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-05-08T15:59:42.683 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T15:59:42.685+0000 7f5ad14f77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T15:59:42.684 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T15:59:42.685+0000 7f5ad14f77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T15:59:42.686 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T15:59:42.689+0000 7f5ad14f77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T15:59:43.101 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-05-08T15:59:43.102 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-08T15:59:43.102 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-05-08T15:59:43.102 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-08T15:59:43.102 INFO:tasks.workunit.client.0.smithi172.stdout:0 2024-05-08T15:59:43.103 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-08T15:59:43.103 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T15:59:43.103 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-08T15:59:43.103 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T15:59:43.103 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T15:59:43.246 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T15:59:43.249+0000 7f5ad14f77c0 -1 Falling back to public interface 2024-05-08T15:59:43.504 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T15:59:44.505 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T15:59:44.505 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T15:59:44.506 INFO:tasks.workunit.client.0.smithi172.stdout:1 2024-05-08T15:59:44.506 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-08T15:59:44.506 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T15:59:44.506 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T15:59:44.907 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T15:59:45.904 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T15:59:45.904+0000 7f5ad14f77c0 -1 osd.0 0 log_to_monitors true 2024-05-08T15:59:45.909 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T15:59:45.909 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T15:59:45.909 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-08T15:59:45.909 INFO:tasks.workunit.client.0.smithi172.stdout:2 2024-05-08T15:59:45.910 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T15:59:45.910 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T15:59:46.399 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T15:59:47.401 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T15:59:47.401 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T15:59:47.401 INFO:tasks.workunit.client.0.smithi172.stdout:3 2024-05-08T15:59:47.402 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-05-08T15:59:47.402 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T15:59:47.402 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T15:59:47.909 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T15:59:48.910 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T15:59:48.911 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T15:59:48.911 INFO:tasks.workunit.client.0.smithi172.stdout:4 2024-05-08T15:59:48.911 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-05-08T15:59:48.912 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T15:59:48.912 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T15:59:49.340 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T15:59:50.341 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T15:59:50.341 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T15:59:50.342 INFO:tasks.workunit.client.0.smithi172.stdout:5 2024-05-08T15:59:50.342 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-05-08T15:59:50.342 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T15:59:50.342 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T15:59:50.854 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T15:59:51.856 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T15:59:51.857 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T15:59:51.857 INFO:tasks.workunit.client.0.smithi172.stdout:6 2024-05-08T15:59:51.857 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-05-08T15:59:51.857 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T15:59:51.857 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T15:59:52.327 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T15:59:53.329 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T15:59:53.329 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T15:59:53.329 INFO:tasks.workunit.client.0.smithi172.stdout:7 2024-05-08T15:59:53.330 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-05-08T15:59:53.330 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T15:59:53.330 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T15:59:53.820 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T15:59:54.823 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T15:59:54.823 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T15:59:54.823 INFO:tasks.workunit.client.0.smithi172.stdout:8 2024-05-08T15:59:54.823 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-05-08T15:59:54.824 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T15:59:54.824 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T15:59:55.280 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T15:59:56.282 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T15:59:56.282 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T15:59:56.282 INFO:tasks.workunit.client.0.smithi172.stdout:9 2024-05-08T15:59:56.282 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-05-08T15:59:56.283 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T15:59:56.283 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T15:59:56.747 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T15:59:57.748 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T15:59:57.748 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T15:59:57.749 INFO:tasks.workunit.client.0.smithi172.stdout:10 2024-05-08T15:59:57.749 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-05-08T15:59:57.749 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T15:59:57.749 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T15:59:58.180 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T15:59:59.182 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T15:59:59.182 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T15:59:59.182 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-05-08T15:59:59.182 INFO:tasks.workunit.client.0.smithi172.stdout:11 2024-05-08T15:59:59.183 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T15:59:59.183 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T15:59:59.773 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:00:00.774 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:00:00.774 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:00:00.775 INFO:tasks.workunit.client.0.smithi172.stdout:12 2024-05-08T16:00:00.775 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-05-08T16:00:00.775 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:00:00.775 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:00:01.624 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:00:02.626 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:00:02.626 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:00:02.627 INFO:tasks.workunit.client.0.smithi172.stdout:13 2024-05-08T16:00:02.627 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-05-08T16:00:02.627 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:00:02.627 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:00:03.756 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:00:04.761 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:00:04.761 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:00:04.762 INFO:tasks.workunit.client.0.smithi172.stdout:14 2024-05-08T16:00:04.762 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-05-08T16:00:04.762 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:00:04.762 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:00:05.295 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:00:06.297 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:00:06.297 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:00:06.298 INFO:tasks.workunit.client.0.smithi172.stdout:15 2024-05-08T16:00:06.298 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-05-08T16:00:06.298 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:00:06.298 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:00:07.422 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:00:08.423 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:00:08.424 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:00:08.424 INFO:tasks.workunit.client.0.smithi172.stdout:16 2024-05-08T16:00:08.424 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-05-08T16:00:08.424 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:00:08.425 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:00:09.161 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:00:10.162 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:00:10.162 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:00:10.162 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-05-08T16:00:10.162 INFO:tasks.workunit.client.0.smithi172.stdout:17 2024-05-08T16:00:10.163 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:00:10.163 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:00:10.711 INFO:tasks.workunit.client.0.smithi172.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3716268245,v1:127.0.0.1:6803/3716268245] [v2:127.0.0.1:6804/3716268245,v1:127.0.0.1:6805/3716268245] exists,up 1db2c695-418f-426b-ae35-c3b454f418d3 2024-05-08T16:00:10.711 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-08T16:00:10.712 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-08T16:00:10.712 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-08T16:00:10.712 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:66: TEST_preload_no_warning: get_asok_path osd.0 2024-05-08T16:00:10.712 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-05-08T16:00:10.712 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-05-08T16:00:10.713 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-08T16:00:10.713 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T16:00:10.713 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20299 2024-05-08T16:00:10.713 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20299/ceph-osd.0.asok 2024-05-08T16:00:10.713 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:66: TEST_preload_no_warning: CEPH_ARGS= 2024-05-08T16:00:10.714 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:66: TEST_preload_no_warning: ceph --admin-daemon /tmp/ceph-asok.20299/ceph-osd.0.asok log flush 2024-05-08T16:00:10.817 INFO:tasks.workunit.client.0.smithi172.stdout:{} 2024-05-08T16:00:10.829 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:67: TEST_preload_no_warning: grep 'WARNING: osd_erasure_code_plugins contains plugin' td/test-erasure-code-plugins/mon.a.log 2024-05-08T16:00:10.834 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:68: TEST_preload_no_warning: grep 'WARNING: osd_erasure_code_plugins contains plugin' td/test-erasure-code-plugins/osd.0.log 2024-05-08T16:00:10.839 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:69: TEST_preload_no_warning: teardown td/test-erasure-code-plugins 2024-05-08T16:00:10.839 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-code-plugins 2024-05-08T16:00:10.840 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-05-08T16:00:10.840 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-code-plugins KILL 2024-05-08T16:00:10.840 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-08T16:00:10.840 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-08T16:00:10.840 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-08T16:00:10.841 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-08T16:00:10.841 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-08T16:00:10.947 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-08T16:00:10.947 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-05-08T16:00:10.948 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-05-08T16:00:10.948 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-05-08T16:00:10.949 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-05-08T16:00:10.949 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-05-08T16:00:10.949 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-05-08T16:00:10.950 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-08T16:00:10.950 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-05-08T16:00:10.950 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-05-08T16:00:10.950 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-08T16:00:10.951 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-05-08T16:00:10.952 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-05-08T16:00:10.952 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-code-plugins 2024-05-08T16:00:10.957 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-05-08T16:00:10.957 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T16:00:10.958 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20299 2024-05-08T16:00:10.958 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.20299 2024-05-08T16:00:10.959 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-05-08T16:00:10.959 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-05-08T16:00:10.959 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:60: TEST_preload_no_warning: for plugin in ${plugins[*]} 2024-05-08T16:00:10.959 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:61: TEST_preload_no_warning: setup td/test-erasure-code-plugins 2024-05-08T16:00:10.960 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/test-erasure-code-plugins 2024-05-08T16:00:10.960 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/test-erasure-code-plugins 2024-05-08T16:00:10.960 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-code-plugins 2024-05-08T16:00:10.960 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-05-08T16:00:10.960 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-code-plugins KILL 2024-05-08T16:00:10.960 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-08T16:00:10.960 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-08T16:00:10.960 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-08T16:00:10.960 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-08T16:00:10.960 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-08T16:00:10.961 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-08T16:00:10.961 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-05-08T16:00:10.961 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-05-08T16:00:10.962 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-05-08T16:00:10.963 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-05-08T16:00:10.963 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-05-08T16:00:10.963 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-05-08T16:00:10.964 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-08T16:00:10.964 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-05-08T16:00:10.965 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-05-08T16:00:10.965 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-08T16:00:10.965 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-05-08T16:00:10.966 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-05-08T16:00:10.966 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-code-plugins 2024-05-08T16:00:10.967 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-05-08T16:00:10.968 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T16:00:10.968 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20299 2024-05-08T16:00:10.968 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.20299 2024-05-08T16:00:10.968 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-05-08T16:00:10.969 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-05-08T16:00:10.969 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/test-erasure-code-plugins 2024-05-08T16:00:10.970 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-05-08T16:00:10.970 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T16:00:10.970 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20299 2024-05-08T16:00:10.970 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.20299 2024-05-08T16:00:10.972 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-05-08T16:00:10.973 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-05-08T16:00:10.973 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-05-08T16:00:10.973 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/test-erasure-code-plugins 1' TERM HUP INT 2024-05-08T16:00:10.973 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:62: TEST_preload_no_warning: run_mon td/test-erasure-code-plugins a --osd_erasure_code_plugins=isa 2024-05-08T16:00:10.973 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/test-erasure-code-plugins 2024-05-08T16:00:10.973 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-05-08T16:00:10.973 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-05-08T16:00:10.973 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-05-08T16:00:10.973 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/test-erasure-code-plugins/a 2024-05-08T16:00:10.973 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-code-plugins/a --run-dir=td/test-erasure-code-plugins --osd_erasure_code_plugins=isa 2024-05-08T16:00:11.388 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-05-08T16:00:11.389 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-08T16:00:11.389 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-08T16:00:11.389 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-08T16:00:11.389 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T16:00:11.389 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20299 2024-05-08T16:00:11.389 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.20299/$cluster-$name.asok' 2024-05-08T16:00:11.390 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-code-plugins/a '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.20299/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-code-plugins/log --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_erasure_code_plugins=isa 2024-05-08T16:00:11.600 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-05-08T16:00:11.601 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-05-08T16:00:11.601 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-05-08T16:00:11.601 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-05-08T16:00:11.601 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-05-08T16:00:11.602 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-05-08T16:00:11.602 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-05-08T16:00:11.602 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-05-08T16:00:11.603 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-05-08T16:00:11.603 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-08T16:00:11.603 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T16:00:11.604 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20299 2024-05-08T16:00:11.604 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20299/ceph-mon.a.asok 2024-05-08T16:00:11.604 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-05-08T16:00:11.604 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.20299/ceph-mon.a.asok config get fsid 2024-05-08T16:00:11.716 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-05-08T16:00:11.716 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-05-08T16:00:11.717 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-05-08T16:00:11.717 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-05-08T16:00:11.717 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-05-08T16:00:11.718 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-05-08T16:00:11.718 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-05-08T16:00:11.718 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-05-08T16:00:11.718 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-08T16:00:11.718 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T16:00:11.718 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20299 2024-05-08T16:00:11.718 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20299/ceph-mon.a.asok 2024-05-08T16:00:11.718 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-05-08T16:00:11.718 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.20299/ceph-mon.a.asok config get mon_host 2024-05-08T16:00:11.831 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:63: TEST_preload_no_warning: run_mgr td/test-erasure-code-plugins x 2024-05-08T16:00:11.831 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/test-erasure-code-plugins 2024-05-08T16:00:11.831 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-05-08T16:00:11.831 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-05-08T16:00:11.831 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-05-08T16:00:11.831 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/test-erasure-code-plugins/x 2024-05-08T16:00:11.832 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-05-08T16:00:12.730 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-05-08T16:00:12.730 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-08T16:00:12.730 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-08T16:00:12.730 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-08T16:00:12.730 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T16:00:12.730 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20299 2024-05-08T16:00:12.730 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.20299/$cluster-$name.asok' 2024-05-08T16:00:12.731 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-05-08T16:00:12.732 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-code-plugins/x '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.20299/$cluster-$name.asok' --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-05-08T16:00:12.762 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:64: TEST_preload_no_warning: get_asok_path mon.a 2024-05-08T16:00:12.762 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-05-08T16:00:12.762 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-05-08T16:00:12.762 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-08T16:00:12.762 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T16:00:12.762 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20299 2024-05-08T16:00:12.762 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20299/ceph-mon.a.asok 2024-05-08T16:00:12.763 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:64: TEST_preload_no_warning: CEPH_ARGS= 2024-05-08T16:00:12.763 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:64: TEST_preload_no_warning: ceph --admin-daemon /tmp/ceph-asok.20299/ceph-mon.a.asok log flush 2024-05-08T16:00:12.855 INFO:tasks.workunit.client.0.smithi172.stdout:{} 2024-05-08T16:00:12.868 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:65: TEST_preload_no_warning: run_osd td/test-erasure-code-plugins 0 --osd_erasure_code_plugins=isa 2024-05-08T16:00:12.869 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-code-plugins 2024-05-08T16:00:12.869 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-05-08T16:00:12.869 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-05-08T16:00:12.869 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-05-08T16:00:12.869 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-code-plugins/0 2024-05-08T16:00:12.869 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=2a3a20f9-46fd-4090-ba60-6d102331c696 --auth-supported=none --mon-host=127.0.0.1:17110 ' 2024-05-08T16:00:12.869 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-08T16:00:12.869 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-05-08T16:00:12.869 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-08T16:00:12.869 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-code-plugins/0' 2024-05-08T16:00:12.869 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code-plugins/0/journal' 2024-05-08T16:00:12.869 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-05-08T16:00:12.869 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-05-08T16:00:12.870 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-code-plugins' 2024-05-08T16:00:12.870 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-05-08T16:00:12.870 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-08T16:00:12.870 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-08T16:00:12.872 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-08T16:00:12.872 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T16:00:12.872 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20299 2024-05-08T16:00:12.872 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.20299/$cluster-$name.asok' 2024-05-08T16:00:12.872 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.20299/$cluster-$name.asok' 2024-05-08T16:00:12.872 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-05-08T16:00:12.872 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-05-08T16:00:12.872 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-05-08T16:00:12.872 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-code-plugins/$name.log' 2024-05-08T16:00:12.873 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-code-plugins/$name.pid' 2024-05-08T16:00:12.873 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-08T16:00:12.873 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-08T16:00:12.873 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-08T16:00:12.873 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-08T16:00:12.873 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-05-08T16:00:12.873 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_erasure_code_plugins=isa 2024-05-08T16:00:12.873 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-code-plugins/0 2024-05-08T16:00:12.873 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-05-08T16:00:12.873 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=e9e57d43-035b-4422-8e97-bcde8a45d903 2024-05-08T16:00:12.874 INFO:tasks.workunit.client.0.smithi172.stdout:add osd0 e9e57d43-035b-4422-8e97-bcde8a45d903 2024-05-08T16:00:12.874 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 e9e57d43-035b-4422-8e97-bcde8a45d903' 2024-05-08T16:00:12.874 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-05-08T16:00:12.889 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCMoTtmwzNMNRAAzABbfX1U31JfomOcchiyoQ== 2024-05-08T16:00:12.889 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCMoTtmwzNMNRAAzABbfX1U31JfomOcchiyoQ=="}' 2024-05-08T16:00:12.889 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new e9e57d43-035b-4422-8e97-bcde8a45d903 -i td/test-erasure-code-plugins/0/new.json 2024-05-08T16:00:13.180 INFO:tasks.workunit.client.0.smithi172.stdout:0 2024-05-08T16:00:13.196 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-code-plugins/0/new.json 2024-05-08T16:00:13.198 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=2a3a20f9-46fd-4090-ba60-6d102331c696 --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.20299/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_erasure_code_plugins=isa --mkfs --key AQCMoTtmwzNMNRAAzABbfX1U31JfomOcchiyoQ== --osd-uuid e9e57d43-035b-4422-8e97-bcde8a45d903 2024-05-08T16:00:13.222 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:00:13.224+0000 7fbcf021d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:00:13.240 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:00:13.240+0000 7fbcf021d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:00:13.241 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:00:13.244+0000 7fbcf021d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:00:13.241 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:00:13.244+0000 7fbcf021d7c0 -1 bluestore(td/test-erasure-code-plugins/0/block) _read_bdev_label failed to open td/test-erasure-code-plugins/0/block: (2) No such file or directory 2024-05-08T16:00:13.241 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:00:13.244+0000 7fbcf021d7c0 -1 bluestore(td/test-erasure-code-plugins/0/block) _read_bdev_label failed to open td/test-erasure-code-plugins/0/block: (2) No such file or directory 2024-05-08T16:00:13.242 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:00:13.244+0000 7fbcf021d7c0 -1 bluestore(td/test-erasure-code-plugins/0/block) _read_bdev_label failed to open td/test-erasure-code-plugins/0/block: (2) No such file or directory 2024-05-08T16:00:13.272 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:00:13.272+0000 7fbcf021d7c0 -1 bluestore(td/test-erasure-code-plugins/0) _read_fsid unparsable uuid 2024-05-08T16:00:19.436 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-code-plugins/0/keyring 2024-05-08T16:00:19.436 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-05-08T16:00:19.436 INFO:tasks.workunit.client.0.smithi172.stdout:adding osd0 key to auth repository 2024-05-08T16:00:19.437 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-05-08T16:00:19.437 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-code-plugins/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-05-08T16:00:19.972 INFO:tasks.workunit.client.0.smithi172.stdout:start osd.0 2024-05-08T16:00:19.973 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-05-08T16:00:19.973 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-05-08T16:00:19.973 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=2a3a20f9-46fd-4090-ba60-6d102331c696 --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.20299/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_erasure_code_plugins=isa 2024-05-08T16:00:19.973 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-05-08T16:00:19.974 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-05-08T16:00:20.001 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:00:20.004+0000 7f993b5467c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:00:20.003 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:00:20.004+0000 7f993b5467c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:00:20.005 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:00:20.008+0000 7f993b5467c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:00:20.381 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-05-08T16:00:20.381 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-08T16:00:20.381 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-05-08T16:00:20.381 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-08T16:00:20.381 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-08T16:00:20.381 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:00:20.381 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-08T16:00:20.382 INFO:tasks.workunit.client.0.smithi172.stdout:0 2024-05-08T16:00:20.382 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:00:20.382 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:00:20.558 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:00:20.560+0000 7f993b5467c0 -1 Falling back to public interface 2024-05-08T16:00:20.796 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:00:21.797 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:00:21.797 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:00:21.797 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-08T16:00:21.797 INFO:tasks.workunit.client.0.smithi172.stdout:1 2024-05-08T16:00:21.798 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:00:21.798 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:00:22.200 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:00:23.141 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:00:23.144+0000 7f993b5467c0 -1 osd.0 0 log_to_monitors true 2024-05-08T16:00:23.202 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:00:23.202 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:00:23.202 INFO:tasks.workunit.client.0.smithi172.stdout:2 2024-05-08T16:00:23.203 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-08T16:00:23.203 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:00:23.203 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:00:23.873 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:00:24.873 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:00:24.873 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:00:24.873 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-05-08T16:00:24.874 INFO:tasks.workunit.client.0.smithi172.stdout:3 2024-05-08T16:00:24.874 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:00:24.874 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:00:25.290 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:00:26.292 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:00:26.292 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:00:26.292 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-05-08T16:00:26.293 INFO:tasks.workunit.client.0.smithi172.stdout:4 2024-05-08T16:00:26.293 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:00:26.293 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:00:26.705 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:00:27.707 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:00:27.723 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:00:27.724 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-05-08T16:00:27.724 INFO:tasks.workunit.client.0.smithi172.stdout:5 2024-05-08T16:00:27.724 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:00:27.725 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:00:28.212 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:00:29.213 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:00:29.214 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:00:29.214 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-05-08T16:00:29.214 INFO:tasks.workunit.client.0.smithi172.stdout:6 2024-05-08T16:00:29.215 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:00:29.215 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:00:29.615 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:00:30.617 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:00:30.617 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:00:30.617 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-05-08T16:00:30.617 INFO:tasks.workunit.client.0.smithi172.stdout:7 2024-05-08T16:00:30.618 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:00:30.618 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:00:31.132 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:00:32.134 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:00:32.134 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:00:32.134 INFO:tasks.workunit.client.0.smithi172.stdout:8 2024-05-08T16:00:32.135 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-05-08T16:00:32.135 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:00:32.135 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:00:32.611 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:00:33.613 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:00:33.613 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:00:33.613 INFO:tasks.workunit.client.0.smithi172.stdout:9 2024-05-08T16:00:33.614 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-05-08T16:00:33.614 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:00:33.614 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:00:34.090 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:00:35.092 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:00:35.092 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:00:35.092 INFO:tasks.workunit.client.0.smithi172.stdout:10 2024-05-08T16:00:35.093 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-05-08T16:00:35.093 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:00:35.093 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:00:35.538 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:00:36.540 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:00:36.540 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:00:36.540 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-05-08T16:00:36.540 INFO:tasks.workunit.client.0.smithi172.stdout:11 2024-05-08T16:00:36.541 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:00:36.541 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:00:37.352 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:00:38.353 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:00:38.354 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:00:38.354 INFO:tasks.workunit.client.0.smithi172.stdout:12 2024-05-08T16:00:38.354 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-05-08T16:00:38.355 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:00:38.355 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:00:39.184 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:00:40.187 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:00:40.187 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:00:40.187 INFO:tasks.workunit.client.0.smithi172.stdout:13 2024-05-08T16:00:40.188 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-05-08T16:00:40.188 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:00:40.188 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:00:41.061 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:00:42.062 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:00:42.063 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:00:42.063 INFO:tasks.workunit.client.0.smithi172.stdout:14 2024-05-08T16:00:42.063 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-05-08T16:00:42.063 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:00:42.064 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:00:43.097 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:00:44.099 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:00:44.099 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:00:44.099 INFO:tasks.workunit.client.0.smithi172.stdout:15 2024-05-08T16:00:44.100 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-05-08T16:00:44.100 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:00:44.100 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:00:44.811 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:00:45.813 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:00:45.813 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:00:45.814 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-05-08T16:00:45.814 INFO:tasks.workunit.client.0.smithi172.stdout:16 2024-05-08T16:00:45.814 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:00:45.814 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:00:46.996 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:00:47.998 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:00:47.998 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:00:47.998 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-05-08T16:00:47.998 INFO:tasks.workunit.client.0.smithi172.stdout:17 2024-05-08T16:00:47.998 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:00:47.999 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:00:48.504 INFO:tasks.workunit.client.0.smithi172.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/975430729,v1:127.0.0.1:6803/975430729] [v2:127.0.0.1:6804/975430729,v1:127.0.0.1:6805/975430729] exists,up e9e57d43-035b-4422-8e97-bcde8a45d903 2024-05-08T16:00:48.505 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-08T16:00:48.505 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-08T16:00:48.505 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-08T16:00:48.505 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:66: TEST_preload_no_warning: get_asok_path osd.0 2024-05-08T16:00:48.505 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-05-08T16:00:48.506 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-05-08T16:00:48.506 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-08T16:00:48.506 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T16:00:48.507 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20299 2024-05-08T16:00:48.507 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20299/ceph-osd.0.asok 2024-05-08T16:00:48.507 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:66: TEST_preload_no_warning: CEPH_ARGS= 2024-05-08T16:00:48.507 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:66: TEST_preload_no_warning: ceph --admin-daemon /tmp/ceph-asok.20299/ceph-osd.0.asok log flush 2024-05-08T16:00:48.618 INFO:tasks.workunit.client.0.smithi172.stdout:{} 2024-05-08T16:00:48.631 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:67: TEST_preload_no_warning: grep 'WARNING: osd_erasure_code_plugins contains plugin' td/test-erasure-code-plugins/mon.a.log 2024-05-08T16:00:48.634 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:68: TEST_preload_no_warning: grep 'WARNING: osd_erasure_code_plugins contains plugin' td/test-erasure-code-plugins/osd.0.log 2024-05-08T16:00:48.636 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:69: TEST_preload_no_warning: teardown td/test-erasure-code-plugins 2024-05-08T16:00:48.636 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-code-plugins 2024-05-08T16:00:48.636 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-05-08T16:00:48.637 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-code-plugins KILL 2024-05-08T16:00:48.637 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-08T16:00:48.637 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-08T16:00:48.637 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-08T16:00:48.637 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-08T16:00:48.637 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-08T16:00:48.746 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-08T16:00:48.746 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-05-08T16:00:48.747 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-05-08T16:00:48.747 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-05-08T16:00:48.748 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-05-08T16:00:48.748 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-05-08T16:00:48.749 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-05-08T16:00:48.749 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-08T16:00:48.749 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-05-08T16:00:48.750 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-05-08T16:00:48.750 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-08T16:00:48.750 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-05-08T16:00:48.752 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-05-08T16:00:48.752 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-code-plugins 2024-05-08T16:00:48.757 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-05-08T16:00:48.757 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T16:00:48.757 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20299 2024-05-08T16:00:48.757 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.20299 2024-05-08T16:00:48.760 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-05-08T16:00:48.760 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-05-08T16:00:48.760 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:72: TEST_preload_no_warning: return 0 2024-05-08T16:00:48.760 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:35: run: for func in $funcs 2024-05-08T16:00:48.760 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:36: run: TEST_preload_no_warning_default td/test-erasure-code-plugins 2024-05-08T16:00:48.760 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:76: TEST_preload_no_warning_default: local dir=td/test-erasure-code-plugins 2024-05-08T16:00:48.760 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:78: TEST_preload_no_warning_default: setup td/test-erasure-code-plugins 2024-05-08T16:00:48.760 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/test-erasure-code-plugins 2024-05-08T16:00:48.760 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/test-erasure-code-plugins 2024-05-08T16:00:48.760 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-code-plugins 2024-05-08T16:00:48.761 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-05-08T16:00:48.761 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-code-plugins KILL 2024-05-08T16:00:48.761 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-08T16:00:48.761 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-08T16:00:48.761 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-08T16:00:48.761 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-08T16:00:48.761 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-08T16:00:48.762 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-08T16:00:48.762 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-05-08T16:00:48.762 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-05-08T16:00:48.762 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-05-08T16:00:48.763 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-05-08T16:00:48.763 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-05-08T16:00:48.763 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-05-08T16:00:48.764 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-08T16:00:48.764 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-05-08T16:00:48.764 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-05-08T16:00:48.764 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-08T16:00:48.765 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-05-08T16:00:48.766 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-05-08T16:00:48.766 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-code-plugins 2024-05-08T16:00:48.767 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-05-08T16:00:48.767 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T16:00:48.767 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20299 2024-05-08T16:00:48.767 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.20299 2024-05-08T16:00:48.768 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-05-08T16:00:48.768 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-05-08T16:00:48.768 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/test-erasure-code-plugins 2024-05-08T16:00:48.770 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-05-08T16:00:48.770 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T16:00:48.770 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20299 2024-05-08T16:00:48.770 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.20299 2024-05-08T16:00:48.772 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-05-08T16:00:48.772 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-05-08T16:00:48.772 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-05-08T16:00:48.772 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/test-erasure-code-plugins 1' TERM HUP INT 2024-05-08T16:00:48.772 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:79: TEST_preload_no_warning_default: run_mon td/test-erasure-code-plugins a 2024-05-08T16:00:48.772 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/test-erasure-code-plugins 2024-05-08T16:00:48.772 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-05-08T16:00:48.772 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-05-08T16:00:48.772 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-05-08T16:00:48.772 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/test-erasure-code-plugins/a 2024-05-08T16:00:48.772 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-code-plugins/a --run-dir=td/test-erasure-code-plugins 2024-05-08T16:00:49.165 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-05-08T16:00:49.166 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-08T16:00:49.166 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-08T16:00:49.167 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-08T16:00:49.167 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T16:00:49.167 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20299 2024-05-08T16:00:49.167 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.20299/$cluster-$name.asok' 2024-05-08T16:00:49.167 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-code-plugins/a '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.20299/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-code-plugins/log --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-05-08T16:00:49.360 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-05-08T16:00:49.361 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-05-08T16:00:49.362 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-05-08T16:00:49.362 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-05-08T16:00:49.362 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-05-08T16:00:49.363 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-05-08T16:00:49.363 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-05-08T16:00:49.363 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-05-08T16:00:49.363 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-05-08T16:00:49.363 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-08T16:00:49.363 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T16:00:49.363 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20299 2024-05-08T16:00:49.363 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20299/ceph-mon.a.asok 2024-05-08T16:00:49.364 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-05-08T16:00:49.364 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.20299/ceph-mon.a.asok config get fsid 2024-05-08T16:00:49.480 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-05-08T16:00:49.480 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-05-08T16:00:49.480 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-05-08T16:00:49.480 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-05-08T16:00:49.481 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-05-08T16:00:49.481 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-05-08T16:00:49.481 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-05-08T16:00:49.481 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-05-08T16:00:49.481 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-08T16:00:49.481 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T16:00:49.481 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20299 2024-05-08T16:00:49.482 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20299/ceph-mon.a.asok 2024-05-08T16:00:49.482 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-05-08T16:00:49.482 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.20299/ceph-mon.a.asok config get mon_host 2024-05-08T16:00:49.591 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:80: TEST_preload_no_warning_default: get_asok_path mon.a 2024-05-08T16:00:49.591 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-05-08T16:00:49.591 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-05-08T16:00:49.592 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-08T16:00:49.592 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T16:00:49.592 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20299 2024-05-08T16:00:49.592 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20299/ceph-mon.a.asok 2024-05-08T16:00:49.593 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:80: TEST_preload_no_warning_default: CEPH_ARGS= 2024-05-08T16:00:49.593 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:80: TEST_preload_no_warning_default: ceph --admin-daemon /tmp/ceph-asok.20299/ceph-mon.a.asok log flush 2024-05-08T16:00:49.686 INFO:tasks.workunit.client.0.smithi172.stdout:{} 2024-05-08T16:00:49.699 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:81: TEST_preload_no_warning_default: run_mgr td/test-erasure-code-plugins x 2024-05-08T16:00:49.699 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/test-erasure-code-plugins 2024-05-08T16:00:49.699 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-05-08T16:00:49.699 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-05-08T16:00:49.699 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-05-08T16:00:49.700 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/test-erasure-code-plugins/x 2024-05-08T16:00:49.700 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-05-08T16:00:50.593 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-05-08T16:00:50.594 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-08T16:00:50.594 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-08T16:00:50.594 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-08T16:00:50.594 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T16:00:50.594 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20299 2024-05-08T16:00:50.595 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.20299/$cluster-$name.asok' 2024-05-08T16:00:50.595 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-05-08T16:00:50.597 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-code-plugins/x '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.20299/$cluster-$name.asok' --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-05-08T16:00:50.626 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:82: TEST_preload_no_warning_default: run_osd td/test-erasure-code-plugins 0 2024-05-08T16:00:50.626 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-code-plugins 2024-05-08T16:00:50.626 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-05-08T16:00:50.626 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-05-08T16:00:50.626 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-05-08T16:00:50.626 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-code-plugins/0 2024-05-08T16:00:50.626 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=2a3a20f9-46fd-4090-ba60-6d102331c696 --auth-supported=none --mon-host=127.0.0.1:17110 ' 2024-05-08T16:00:50.627 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-08T16:00:50.627 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-05-08T16:00:50.627 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-08T16:00:50.627 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-code-plugins/0' 2024-05-08T16:00:50.627 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code-plugins/0/journal' 2024-05-08T16:00:50.627 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-05-08T16:00:50.627 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-05-08T16:00:50.627 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-code-plugins' 2024-05-08T16:00:50.627 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-05-08T16:00:50.627 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-08T16:00:50.627 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-08T16:00:50.627 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-08T16:00:50.627 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T16:00:50.627 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20299 2024-05-08T16:00:50.627 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.20299/$cluster-$name.asok' 2024-05-08T16:00:50.628 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.20299/$cluster-$name.asok' 2024-05-08T16:00:50.628 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-05-08T16:00:50.628 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-05-08T16:00:50.628 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-05-08T16:00:50.628 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-code-plugins/$name.log' 2024-05-08T16:00:50.628 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-code-plugins/$name.pid' 2024-05-08T16:00:50.628 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-08T16:00:50.628 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-08T16:00:50.628 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-08T16:00:50.628 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-08T16:00:50.628 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-05-08T16:00:50.628 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-05-08T16:00:50.628 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-code-plugins/0 2024-05-08T16:00:50.628 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-05-08T16:00:50.629 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=274e3897-7648-4b99-9883-1bc1fa09383a 2024-05-08T16:00:50.629 INFO:tasks.workunit.client.0.smithi172.stdout:add osd0 274e3897-7648-4b99-9883-1bc1fa09383a 2024-05-08T16:00:50.629 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 274e3897-7648-4b99-9883-1bc1fa09383a' 2024-05-08T16:00:50.630 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-05-08T16:00:50.643 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCyoTtmIQa5JhAATNnrRZy3QoBsQHkKnur8MA== 2024-05-08T16:00:50.644 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCyoTtmIQa5JhAATNnrRZy3QoBsQHkKnur8MA=="}' 2024-05-08T16:00:50.644 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 274e3897-7648-4b99-9883-1bc1fa09383a -i td/test-erasure-code-plugins/0/new.json 2024-05-08T16:00:50.937 INFO:tasks.workunit.client.0.smithi172.stdout:0 2024-05-08T16:00:50.953 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-code-plugins/0/new.json 2024-05-08T16:00:50.954 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=2a3a20f9-46fd-4090-ba60-6d102331c696 --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.20299/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCyoTtmIQa5JhAATNnrRZy3QoBsQHkKnur8MA== --osd-uuid 274e3897-7648-4b99-9883-1bc1fa09383a 2024-05-08T16:00:50.991 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:00:50.992+0000 7f93ab9107c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:00:51.006 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:00:51.008+0000 7f93ab9107c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:00:51.008 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:00:51.008+0000 7f93ab9107c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:00:51.008 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:00:51.012+0000 7f93ab9107c0 -1 bluestore(td/test-erasure-code-plugins/0/block) _read_bdev_label failed to open td/test-erasure-code-plugins/0/block: (2) No such file or directory 2024-05-08T16:00:51.008 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:00:51.012+0000 7f93ab9107c0 -1 bluestore(td/test-erasure-code-plugins/0/block) _read_bdev_label failed to open td/test-erasure-code-plugins/0/block: (2) No such file or directory 2024-05-08T16:00:51.008 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:00:51.012+0000 7f93ab9107c0 -1 bluestore(td/test-erasure-code-plugins/0/block) _read_bdev_label failed to open td/test-erasure-code-plugins/0/block: (2) No such file or directory 2024-05-08T16:00:51.038 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:00:51.040+0000 7f93ab9107c0 -1 bluestore(td/test-erasure-code-plugins/0) _read_fsid unparsable uuid 2024-05-08T16:00:57.265 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-code-plugins/0/keyring 2024-05-08T16:00:57.265 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-05-08T16:00:57.266 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-05-08T16:00:57.266 INFO:tasks.workunit.client.0.smithi172.stdout:adding osd0 key to auth repository 2024-05-08T16:00:57.267 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-code-plugins/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-05-08T16:00:57.827 INFO:tasks.workunit.client.0.smithi172.stdout:start osd.0 2024-05-08T16:00:57.828 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-05-08T16:00:57.828 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=2a3a20f9-46fd-4090-ba60-6d102331c696 --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.20299/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-05-08T16:00:57.828 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-05-08T16:00:57.828 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-05-08T16:00:57.828 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-05-08T16:00:57.856 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:00:57.860+0000 7fcb19fcb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:00:57.859 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:00:57.860+0000 7fcb19fcb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:00:57.860 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:00:57.860+0000 7fcb19fcb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:00:58.212 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-05-08T16:00:58.212 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-08T16:00:58.212 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-05-08T16:00:58.212 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-08T16:00:58.212 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-08T16:00:58.212 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:00:58.213 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-08T16:00:58.213 INFO:tasks.workunit.client.0.smithi172.stdout:0 2024-05-08T16:00:58.213 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:00:58.213 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:00:58.406 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:00:58.408+0000 7fcb19fcb7c0 -1 Falling back to public interface 2024-05-08T16:00:58.630 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:00:59.632 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:00:59.632 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:00:59.632 INFO:tasks.workunit.client.0.smithi172.stdout:1 2024-05-08T16:00:59.633 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-08T16:00:59.633 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:00:59.633 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:01:00.023 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:01:01.024 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:01:01.025 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:01:01.025 INFO:tasks.workunit.client.0.smithi172.stdout:2 2024-05-08T16:01:01.025 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-08T16:01:01.025 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:01:01.026 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:01:01.027 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:01:01.028+0000 7fcb19fcb7c0 -1 osd.0 0 log_to_monitors true 2024-05-08T16:01:01.556 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:01:02.557 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:01:02.557 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:01:02.558 INFO:tasks.workunit.client.0.smithi172.stdout:3 2024-05-08T16:01:02.558 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-05-08T16:01:02.558 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:01:02.558 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:01:03.016 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:01:04.018 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:01:04.018 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:01:04.018 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-05-08T16:01:04.019 INFO:tasks.workunit.client.0.smithi172.stdout:4 2024-05-08T16:01:04.019 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:01:04.019 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:01:04.447 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:01:05.449 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:01:05.449 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:01:05.449 INFO:tasks.workunit.client.0.smithi172.stdout:5 2024-05-08T16:01:05.450 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-05-08T16:01:05.450 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:01:05.450 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:01:05.897 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:01:06.899 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:01:06.899 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:01:06.899 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-05-08T16:01:06.899 INFO:tasks.workunit.client.0.smithi172.stdout:6 2024-05-08T16:01:06.900 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:01:06.900 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:01:07.402 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:01:08.405 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:01:08.405 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:01:08.405 INFO:tasks.workunit.client.0.smithi172.stdout:7 2024-05-08T16:01:08.406 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-05-08T16:01:08.406 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:01:08.406 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:01:08.907 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:01:09.909 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:01:09.909 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:01:09.909 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-05-08T16:01:09.909 INFO:tasks.workunit.client.0.smithi172.stdout:8 2024-05-08T16:01:09.910 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:01:09.910 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:01:10.356 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:01:11.357 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:01:11.357 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:01:11.358 INFO:tasks.workunit.client.0.smithi172.stdout:9 2024-05-08T16:01:11.358 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-05-08T16:01:11.358 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:01:11.358 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:01:11.937 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:01:12.939 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:01:12.939 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:01:12.940 INFO:tasks.workunit.client.0.smithi172.stdout:10 2024-05-08T16:01:12.940 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-05-08T16:01:12.940 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:01:12.940 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:01:13.550 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:01:14.551 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:01:14.551 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:01:14.552 INFO:tasks.workunit.client.0.smithi172.stdout:11 2024-05-08T16:01:14.552 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-05-08T16:01:14.552 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:01:14.552 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:01:15.115 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:01:16.117 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:01:16.117 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:01:16.117 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-05-08T16:01:16.117 INFO:tasks.workunit.client.0.smithi172.stdout:12 2024-05-08T16:01:16.118 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:01:16.118 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:01:16.858 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:01:17.860 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:01:17.861 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:01:17.861 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-05-08T16:01:17.861 INFO:tasks.workunit.client.0.smithi172.stdout:13 2024-05-08T16:01:17.861 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:01:17.861 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:01:18.997 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:01:19.999 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:01:19.999 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:01:19.999 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-05-08T16:01:19.999 INFO:tasks.workunit.client.0.smithi172.stdout:14 2024-05-08T16:01:20.000 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:01:20.000 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:01:20.878 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:01:21.880 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:01:21.880 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:01:21.880 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-05-08T16:01:21.881 INFO:tasks.workunit.client.0.smithi172.stdout:15 2024-05-08T16:01:21.881 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:01:21.881 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:01:22.804 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:01:23.806 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:01:23.806 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:01:23.807 INFO:tasks.workunit.client.0.smithi172.stdout:16 2024-05-08T16:01:23.807 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-05-08T16:01:23.807 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:01:23.807 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:01:24.606 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:01:25.608 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:01:25.608 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:01:25.608 INFO:tasks.workunit.client.0.smithi172.stdout:17 2024-05-08T16:01:25.609 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-05-08T16:01:25.609 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:01:25.609 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:01:26.013 INFO:tasks.workunit.client.0.smithi172.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/41106453,v1:127.0.0.1:6803/41106453] [v2:127.0.0.1:6804/41106453,v1:127.0.0.1:6805/41106453] exists,up 274e3897-7648-4b99-9883-1bc1fa09383a 2024-05-08T16:01:26.014 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-08T16:01:26.014 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-08T16:01:26.014 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-08T16:01:26.014 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:83: TEST_preload_no_warning_default: get_asok_path osd.0 2024-05-08T16:01:26.014 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-05-08T16:01:26.014 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-05-08T16:01:26.015 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-08T16:01:26.015 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T16:01:26.015 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20299 2024-05-08T16:01:26.015 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20299/ceph-osd.0.asok 2024-05-08T16:01:26.015 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:83: TEST_preload_no_warning_default: CEPH_ARGS= 2024-05-08T16:01:26.015 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:83: TEST_preload_no_warning_default: ceph --admin-daemon /tmp/ceph-asok.20299/ceph-osd.0.asok log flush 2024-05-08T16:01:26.125 INFO:tasks.workunit.client.0.smithi172.stdout:{} 2024-05-08T16:01:26.137 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:84: TEST_preload_no_warning_default: grep 'WARNING: osd_erasure_code_plugins' td/test-erasure-code-plugins/mon.a.log 2024-05-08T16:01:26.142 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:85: TEST_preload_no_warning_default: grep 'WARNING: osd_erasure_code_plugins' td/test-erasure-code-plugins/osd.0.log 2024-05-08T16:01:26.145 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:86: TEST_preload_no_warning_default: teardown td/test-erasure-code-plugins 2024-05-08T16:01:26.145 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-code-plugins 2024-05-08T16:01:26.145 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-05-08T16:01:26.145 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-code-plugins KILL 2024-05-08T16:01:26.146 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-08T16:01:26.146 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-08T16:01:26.146 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-08T16:01:26.146 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-08T16:01:26.146 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-08T16:01:26.254 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-08T16:01:26.254 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-05-08T16:01:26.254 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-05-08T16:01:26.255 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-05-08T16:01:26.256 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-05-08T16:01:26.256 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-05-08T16:01:26.256 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-05-08T16:01:26.256 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-08T16:01:26.257 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-05-08T16:01:26.257 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-05-08T16:01:26.257 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-08T16:01:26.258 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-05-08T16:01:26.259 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-05-08T16:01:26.259 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-code-plugins 2024-05-08T16:01:26.264 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-05-08T16:01:26.264 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T16:01:26.264 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20299 2024-05-08T16:01:26.264 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.20299 2024-05-08T16:01:26.267 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-05-08T16:01:26.267 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-05-08T16:01:26.267 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:88: TEST_preload_no_warning_default: return 0 2024-05-08T16:01:26.267 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:35: run: for func in $funcs 2024-05-08T16:01:26.267 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:36: run: TEST_preload_warning td/test-erasure-code-plugins 2024-05-08T16:01:26.267 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:41: TEST_preload_warning: local dir=td/test-erasure-code-plugins 2024-05-08T16:01:26.267 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:43: TEST_preload_warning: for plugin in ${legacy_jerasure_plugins[*]} ${legacy_shec_plugins[*]} 2024-05-08T16:01:26.267 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:44: TEST_preload_warning: setup td/test-erasure-code-plugins 2024-05-08T16:01:26.267 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/test-erasure-code-plugins 2024-05-08T16:01:26.267 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/test-erasure-code-plugins 2024-05-08T16:01:26.267 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-code-plugins 2024-05-08T16:01:26.267 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-05-08T16:01:26.267 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-code-plugins KILL 2024-05-08T16:01:26.268 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-08T16:01:26.268 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-08T16:01:26.268 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-08T16:01:26.268 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-08T16:01:26.268 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-08T16:01:26.269 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-08T16:01:26.269 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-05-08T16:01:26.269 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-05-08T16:01:26.269 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-05-08T16:01:26.269 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-05-08T16:01:26.269 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-05-08T16:01:26.270 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-05-08T16:01:26.270 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-08T16:01:26.270 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-05-08T16:01:26.271 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-05-08T16:01:26.271 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-08T16:01:26.271 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-05-08T16:01:26.272 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-05-08T16:01:26.272 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-code-plugins 2024-05-08T16:01:26.274 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-05-08T16:01:26.274 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T16:01:26.274 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20299 2024-05-08T16:01:26.274 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.20299 2024-05-08T16:01:26.274 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-05-08T16:01:26.275 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-05-08T16:01:26.275 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/test-erasure-code-plugins 2024-05-08T16:01:26.276 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-05-08T16:01:26.276 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T16:01:26.276 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20299 2024-05-08T16:01:26.276 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.20299 2024-05-08T16:01:26.277 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-05-08T16:01:26.277 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-05-08T16:01:26.277 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-05-08T16:01:26.278 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/test-erasure-code-plugins 1' TERM HUP INT 2024-05-08T16:01:26.278 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:45: TEST_preload_warning: run_mon td/test-erasure-code-plugins a --osd_erasure_code_plugins=jerasure_generic 2024-05-08T16:01:26.278 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/test-erasure-code-plugins 2024-05-08T16:01:26.278 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-05-08T16:01:26.278 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-05-08T16:01:26.278 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-05-08T16:01:26.278 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/test-erasure-code-plugins/a 2024-05-08T16:01:26.278 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-code-plugins/a --run-dir=td/test-erasure-code-plugins --osd_erasure_code_plugins=jerasure_generic 2024-05-08T16:01:26.697 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-05-08T16:01:26.697 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-08T16:01:26.697 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-08T16:01:26.697 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-08T16:01:26.697 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T16:01:26.697 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20299 2024-05-08T16:01:26.698 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.20299/$cluster-$name.asok' 2024-05-08T16:01:26.698 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-code-plugins/a '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.20299/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-code-plugins/log --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_erasure_code_plugins=jerasure_generic 2024-05-08T16:01:26.896 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-05-08T16:01:26.897 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-05-08T16:01:26.897 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-05-08T16:01:26.897 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-05-08T16:01:26.897 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-05-08T16:01:26.897 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-05-08T16:01:26.898 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-05-08T16:01:26.898 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-05-08T16:01:26.898 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-05-08T16:01:26.899 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-08T16:01:26.899 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T16:01:26.899 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20299 2024-05-08T16:01:26.899 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20299/ceph-mon.a.asok 2024-05-08T16:01:26.899 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-05-08T16:01:26.899 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.20299/ceph-mon.a.asok config get fsid 2024-05-08T16:01:27.015 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-05-08T16:01:27.015 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-05-08T16:01:27.016 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-05-08T16:01:27.016 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-05-08T16:01:27.017 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-05-08T16:01:27.017 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-05-08T16:01:27.017 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-05-08T16:01:27.017 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-05-08T16:01:27.017 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-08T16:01:27.017 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T16:01:27.017 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20299 2024-05-08T16:01:27.017 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20299/ceph-mon.a.asok 2024-05-08T16:01:27.018 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-05-08T16:01:27.018 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.20299/ceph-mon.a.asok config get mon_host 2024-05-08T16:01:27.126 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:46: TEST_preload_warning: run_mgr td/test-erasure-code-plugins x 2024-05-08T16:01:27.126 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/test-erasure-code-plugins 2024-05-08T16:01:27.127 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-05-08T16:01:27.127 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-05-08T16:01:27.127 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-05-08T16:01:27.127 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/test-erasure-code-plugins/x 2024-05-08T16:01:27.127 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-05-08T16:01:28.045 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-05-08T16:01:28.045 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-08T16:01:28.045 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-08T16:01:28.045 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-08T16:01:28.045 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T16:01:28.046 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20299 2024-05-08T16:01:28.046 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.20299/$cluster-$name.asok' 2024-05-08T16:01:28.046 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-05-08T16:01:28.046 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-code-plugins/x '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.20299/$cluster-$name.asok' --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-05-08T16:01:28.079 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:47: TEST_preload_warning: get_asok_path mon.a 2024-05-08T16:01:28.079 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-05-08T16:01:28.079 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-05-08T16:01:28.079 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-08T16:01:28.080 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T16:01:28.080 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20299 2024-05-08T16:01:28.080 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20299/ceph-mon.a.asok 2024-05-08T16:01:28.080 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:47: TEST_preload_warning: CEPH_ARGS= 2024-05-08T16:01:28.080 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:47: TEST_preload_warning: ceph --admin-daemon /tmp/ceph-asok.20299/ceph-mon.a.asok log flush 2024-05-08T16:01:28.178 INFO:tasks.workunit.client.0.smithi172.stdout:{} 2024-05-08T16:01:28.193 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:48: TEST_preload_warning: run_osd td/test-erasure-code-plugins 0 --osd_erasure_code_plugins=jerasure_generic 2024-05-08T16:01:28.193 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-code-plugins 2024-05-08T16:01:28.194 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-05-08T16:01:28.194 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-05-08T16:01:28.194 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-05-08T16:01:28.194 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-code-plugins/0 2024-05-08T16:01:28.194 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=2a3a20f9-46fd-4090-ba60-6d102331c696 --auth-supported=none --mon-host=127.0.0.1:17110 ' 2024-05-08T16:01:28.194 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-08T16:01:28.194 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-05-08T16:01:28.194 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-08T16:01:28.194 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-code-plugins/0' 2024-05-08T16:01:28.194 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code-plugins/0/journal' 2024-05-08T16:01:28.194 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-05-08T16:01:28.194 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-05-08T16:01:28.195 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-code-plugins' 2024-05-08T16:01:28.195 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-05-08T16:01:28.195 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-08T16:01:28.195 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-08T16:01:28.196 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-08T16:01:28.197 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T16:01:28.197 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20299 2024-05-08T16:01:28.197 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.20299/$cluster-$name.asok' 2024-05-08T16:01:28.197 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.20299/$cluster-$name.asok' 2024-05-08T16:01:28.197 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-05-08T16:01:28.197 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-05-08T16:01:28.197 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-05-08T16:01:28.197 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-code-plugins/$name.log' 2024-05-08T16:01:28.197 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-code-plugins/$name.pid' 2024-05-08T16:01:28.197 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-08T16:01:28.197 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-08T16:01:28.197 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-08T16:01:28.197 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-08T16:01:28.198 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-05-08T16:01:28.198 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_erasure_code_plugins=jerasure_generic 2024-05-08T16:01:28.198 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-code-plugins/0 2024-05-08T16:01:28.198 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-05-08T16:01:28.198 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=c8db6f8d-554a-473c-952f-0be0a3d202ef 2024-05-08T16:01:28.198 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 c8db6f8d-554a-473c-952f-0be0a3d202ef' 2024-05-08T16:01:28.199 INFO:tasks.workunit.client.0.smithi172.stdout:add osd0 c8db6f8d-554a-473c-952f-0be0a3d202ef 2024-05-08T16:01:28.199 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-05-08T16:01:28.214 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDYoTtm7RgmDRAA9mDNRM7ashlfgP/Zsyz6Ag== 2024-05-08T16:01:28.214 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDYoTtm7RgmDRAA9mDNRM7ashlfgP/Zsyz6Ag=="}' 2024-05-08T16:01:28.214 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new c8db6f8d-554a-473c-952f-0be0a3d202ef -i td/test-erasure-code-plugins/0/new.json 2024-05-08T16:01:28.536 INFO:tasks.workunit.client.0.smithi172.stdout:0 2024-05-08T16:01:28.552 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-code-plugins/0/new.json 2024-05-08T16:01:28.552 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=2a3a20f9-46fd-4090-ba60-6d102331c696 --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.20299/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_erasure_code_plugins=jerasure_generic --mkfs --key AQDYoTtm7RgmDRAA9mDNRM7ashlfgP/Zsyz6Ag== --osd-uuid c8db6f8d-554a-473c-952f-0be0a3d202ef 2024-05-08T16:01:28.581 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:01:28.584+0000 7f1f2f61f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:01:28.612 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:01:28.616+0000 7f1f2f61f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:01:28.614 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:01:28.616+0000 7f1f2f61f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:01:28.614 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:01:28.616+0000 7f1f2f61f7c0 -1 bluestore(td/test-erasure-code-plugins/0/block) _read_bdev_label failed to open td/test-erasure-code-plugins/0/block: (2) No such file or directory 2024-05-08T16:01:28.614 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:01:28.616+0000 7f1f2f61f7c0 -1 bluestore(td/test-erasure-code-plugins/0/block) _read_bdev_label failed to open td/test-erasure-code-plugins/0/block: (2) No such file or directory 2024-05-08T16:01:28.614 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:01:28.616+0000 7f1f2f61f7c0 -1 bluestore(td/test-erasure-code-plugins/0/block) _read_bdev_label failed to open td/test-erasure-code-plugins/0/block: (2) No such file or directory 2024-05-08T16:01:28.661 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:01:28.664+0000 7f1f2f61f7c0 -1 bluestore(td/test-erasure-code-plugins/0) _read_fsid unparsable uuid 2024-05-08T16:01:34.849 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-code-plugins/0/keyring 2024-05-08T16:01:34.849 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-05-08T16:01:34.853 INFO:tasks.workunit.client.0.smithi172.stdout:adding osd0 key to auth repository 2024-05-08T16:01:34.853 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-05-08T16:01:34.854 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-code-plugins/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-05-08T16:01:35.400 INFO:tasks.workunit.client.0.smithi172.stdout:start osd.0 2024-05-08T16:01:35.400 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-05-08T16:01:35.401 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=2a3a20f9-46fd-4090-ba60-6d102331c696 --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.20299/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_erasure_code_plugins=jerasure_generic 2024-05-08T16:01:35.401 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-05-08T16:01:35.401 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-05-08T16:01:35.401 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-05-08T16:01:35.431 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:01:35.431+0000 7f70f34bb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:01:35.433 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:01:35.435+0000 7f70f34bb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:01:35.434 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:01:35.435+0000 7f70f34bb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:01:35.829 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-05-08T16:01:35.830 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-08T16:01:35.830 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-05-08T16:01:35.830 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-08T16:01:35.830 INFO:tasks.workunit.client.0.smithi172.stdout:0 2024-05-08T16:01:35.831 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-08T16:01:35.831 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:01:35.831 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-08T16:01:35.831 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:01:35.831 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:01:36.005 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:01:36.007+0000 7f70f34bb7c0 -1 Falling back to public interface 2024-05-08T16:01:36.263 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:01:37.264 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:01:37.264 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:01:37.265 INFO:tasks.workunit.client.0.smithi172.stdout:1 2024-05-08T16:01:37.265 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-08T16:01:37.265 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:01:37.265 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:01:37.660 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:01:38.584 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:01:38.587+0000 7f70f34bb7c0 -1 osd.0 0 log_to_monitors true 2024-05-08T16:01:38.662 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:01:38.662 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:01:38.662 INFO:tasks.workunit.client.0.smithi172.stdout:2 2024-05-08T16:01:38.663 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-08T16:01:38.663 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:01:38.663 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:01:39.227 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:01:40.229 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:01:40.229 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:01:40.229 INFO:tasks.workunit.client.0.smithi172.stdout:3 2024-05-08T16:01:40.229 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-05-08T16:01:40.230 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:01:40.230 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:01:40.673 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:01:41.675 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:01:41.675 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:01:41.675 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-05-08T16:01:41.675 INFO:tasks.workunit.client.0.smithi172.stdout:4 2024-05-08T16:01:41.676 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:01:41.676 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:01:42.122 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:01:43.124 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:01:43.124 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:01:43.124 INFO:tasks.workunit.client.0.smithi172.stdout:5 2024-05-08T16:01:43.125 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-05-08T16:01:43.125 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:01:43.125 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:01:43.570 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:01:44.571 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:01:44.571 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:01:44.572 INFO:tasks.workunit.client.0.smithi172.stdout:6 2024-05-08T16:01:44.573 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-05-08T16:01:44.573 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:01:44.573 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:01:44.999 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:01:46.001 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:01:46.001 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:01:46.001 INFO:tasks.workunit.client.0.smithi172.stdout:7 2024-05-08T16:01:46.002 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-05-08T16:01:46.002 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:01:46.002 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:01:46.556 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:01:47.557 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:01:47.558 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:01:47.558 INFO:tasks.workunit.client.0.smithi172.stdout:8 2024-05-08T16:01:47.558 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-05-08T16:01:47.559 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:01:47.559 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:01:47.988 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:01:48.989 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:01:48.990 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:01:48.990 INFO:tasks.workunit.client.0.smithi172.stdout:9 2024-05-08T16:01:48.991 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-05-08T16:01:48.991 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:01:48.991 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:01:49.536 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:01:50.537 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:01:50.537 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:01:50.538 INFO:tasks.workunit.client.0.smithi172.stdout:10 2024-05-08T16:01:50.538 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-05-08T16:01:50.538 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:01:50.538 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:01:50.991 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:01:51.993 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:01:51.993 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:01:51.993 INFO:tasks.workunit.client.0.smithi172.stdout:11 2024-05-08T16:01:51.994 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-05-08T16:01:51.994 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:01:51.994 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:01:53.105 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:01:54.106 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:01:54.107 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:01:54.107 INFO:tasks.workunit.client.0.smithi172.stdout:12 2024-05-08T16:01:54.107 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-05-08T16:01:54.107 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:01:54.108 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:01:54.953 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:01:55.955 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:01:55.955 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:01:55.955 INFO:tasks.workunit.client.0.smithi172.stdout:13 2024-05-08T16:01:55.956 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-05-08T16:01:55.956 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:01:55.956 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:01:56.829 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:01:57.831 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:01:57.831 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:01:57.831 INFO:tasks.workunit.client.0.smithi172.stdout:14 2024-05-08T16:01:57.832 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-05-08T16:01:57.832 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:01:57.832 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:01:58.704 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:01:59.706 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:01:59.706 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:01:59.706 INFO:tasks.workunit.client.0.smithi172.stdout:15 2024-05-08T16:01:59.707 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-05-08T16:01:59.707 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:01:59.707 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:02:00.629 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:02:01.631 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:02:01.631 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:02:01.632 INFO:tasks.workunit.client.0.smithi172.stdout:16 2024-05-08T16:02:01.632 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-05-08T16:02:01.632 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:02:01.632 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:02:02.748 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:02:03.750 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:02:03.750 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:02:03.750 INFO:tasks.workunit.client.0.smithi172.stdout:17 2024-05-08T16:02:03.751 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-05-08T16:02:03.751 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:02:03.751 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:02:04.168 INFO:tasks.workunit.client.0.smithi172.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/545953920,v1:127.0.0.1:6803/545953920] [v2:127.0.0.1:6804/545953920,v1:127.0.0.1:6805/545953920] exists,up c8db6f8d-554a-473c-952f-0be0a3d202ef 2024-05-08T16:02:04.169 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-08T16:02:04.169 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-08T16:02:04.169 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-08T16:02:04.170 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:49: TEST_preload_warning: get_asok_path osd.0 2024-05-08T16:02:04.170 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-05-08T16:02:04.170 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-05-08T16:02:04.170 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-08T16:02:04.170 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T16:02:04.171 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20299 2024-05-08T16:02:04.171 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20299/ceph-osd.0.asok 2024-05-08T16:02:04.171 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:49: TEST_preload_warning: CEPH_ARGS= 2024-05-08T16:02:04.171 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:49: TEST_preload_warning: ceph --admin-daemon /tmp/ceph-asok.20299/ceph-osd.0.asok log flush 2024-05-08T16:02:04.268 INFO:tasks.workunit.client.0.smithi172.stdout:{} 2024-05-08T16:02:04.279 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:50: TEST_preload_warning: grep 'WARNING: osd_erasure_code_plugins contains plugin jerasure_generic' td/test-erasure-code-plugins/mon.a.log 2024-05-08T16:02:04.282 INFO:tasks.workunit.client.0.smithi172.stdout:2024-05-08T16:01:26.736+0000 7fb69ec02c80 0 WARNING: osd_erasure_code_plugins contains plugin jerasure_generic that is now deprecated. Please modify the value for osd_erasure_code_plugins to use jerasure instead. 2024-05-08T16:02:04.282 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:51: TEST_preload_warning: grep 'WARNING: osd_erasure_code_plugins contains plugin jerasure_generic' td/test-erasure-code-plugins/osd.0.log 2024-05-08T16:02:04.284 INFO:tasks.workunit.client.0.smithi172.stdout:2024-05-08T16:01:36.007+0000 7f70f34bb7c0 0 WARNING: osd_erasure_code_plugins contains plugin jerasure_generic that is now deprecated. Please modify the value for osd_erasure_code_plugins to use jerasure instead. 2024-05-08T16:02:04.284 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:52: TEST_preload_warning: teardown td/test-erasure-code-plugins 2024-05-08T16:02:04.285 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-code-plugins 2024-05-08T16:02:04.285 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-05-08T16:02:04.285 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-code-plugins KILL 2024-05-08T16:02:04.285 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-08T16:02:04.285 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-08T16:02:04.286 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-08T16:02:04.286 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-08T16:02:04.286 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-08T16:02:04.394 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-08T16:02:04.395 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-05-08T16:02:04.396 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-05-08T16:02:04.396 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-05-08T16:02:04.397 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-05-08T16:02:04.397 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-05-08T16:02:04.397 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-05-08T16:02:04.398 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-08T16:02:04.398 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-05-08T16:02:04.398 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-05-08T16:02:04.399 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-08T16:02:04.399 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-05-08T16:02:04.400 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-05-08T16:02:04.400 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-code-plugins 2024-05-08T16:02:04.405 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-05-08T16:02:04.405 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T16:02:04.405 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20299 2024-05-08T16:02:04.406 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.20299 2024-05-08T16:02:04.407 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-05-08T16:02:04.407 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-05-08T16:02:04.407 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:43: TEST_preload_warning: for plugin in ${legacy_jerasure_plugins[*]} ${legacy_shec_plugins[*]} 2024-05-08T16:02:04.407 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:44: TEST_preload_warning: setup td/test-erasure-code-plugins 2024-05-08T16:02:04.408 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/test-erasure-code-plugins 2024-05-08T16:02:04.408 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/test-erasure-code-plugins 2024-05-08T16:02:04.408 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-code-plugins 2024-05-08T16:02:04.408 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-05-08T16:02:04.408 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-code-plugins KILL 2024-05-08T16:02:04.408 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-08T16:02:04.408 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-08T16:02:04.408 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-08T16:02:04.408 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-08T16:02:04.408 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-08T16:02:04.409 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-08T16:02:04.409 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-05-08T16:02:04.410 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-05-08T16:02:04.411 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-05-08T16:02:04.412 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-05-08T16:02:04.412 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-05-08T16:02:04.412 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-05-08T16:02:04.412 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-08T16:02:04.413 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-05-08T16:02:04.413 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-05-08T16:02:04.413 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-08T16:02:04.414 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-05-08T16:02:04.415 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-05-08T16:02:04.415 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-code-plugins 2024-05-08T16:02:04.416 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-05-08T16:02:04.416 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T16:02:04.416 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20299 2024-05-08T16:02:04.416 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.20299 2024-05-08T16:02:04.417 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-05-08T16:02:04.417 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-05-08T16:02:04.417 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/test-erasure-code-plugins 2024-05-08T16:02:04.418 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-05-08T16:02:04.418 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T16:02:04.418 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20299 2024-05-08T16:02:04.418 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.20299 2024-05-08T16:02:04.420 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-05-08T16:02:04.420 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-05-08T16:02:04.420 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-05-08T16:02:04.420 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/test-erasure-code-plugins 1' TERM HUP INT 2024-05-08T16:02:04.421 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:45: TEST_preload_warning: run_mon td/test-erasure-code-plugins a --osd_erasure_code_plugins=jerasure_sse3 2024-05-08T16:02:04.421 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/test-erasure-code-plugins 2024-05-08T16:02:04.421 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-05-08T16:02:04.421 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-05-08T16:02:04.421 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-05-08T16:02:04.421 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/test-erasure-code-plugins/a 2024-05-08T16:02:04.421 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-code-plugins/a --run-dir=td/test-erasure-code-plugins --osd_erasure_code_plugins=jerasure_sse3 2024-05-08T16:02:04.831 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-05-08T16:02:04.832 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-08T16:02:04.832 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-08T16:02:04.832 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-08T16:02:04.832 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T16:02:04.833 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20299 2024-05-08T16:02:04.833 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.20299/$cluster-$name.asok' 2024-05-08T16:02:04.833 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-code-plugins/a '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.20299/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-code-plugins/log --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_erasure_code_plugins=jerasure_sse3 2024-05-08T16:02:05.032 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-05-08T16:02:05.038 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-05-08T16:02:05.038 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-05-08T16:02:05.038 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-05-08T16:02:05.039 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-05-08T16:02:05.039 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-05-08T16:02:05.039 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-05-08T16:02:05.039 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-05-08T16:02:05.039 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-05-08T16:02:05.039 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-08T16:02:05.039 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T16:02:05.039 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20299 2024-05-08T16:02:05.039 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20299/ceph-mon.a.asok 2024-05-08T16:02:05.039 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-05-08T16:02:05.039 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.20299/ceph-mon.a.asok config get fsid 2024-05-08T16:02:05.147 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-05-08T16:02:05.147 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-05-08T16:02:05.147 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-05-08T16:02:05.147 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-05-08T16:02:05.148 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-05-08T16:02:05.148 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-05-08T16:02:05.148 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-05-08T16:02:05.148 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-05-08T16:02:05.149 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-08T16:02:05.149 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T16:02:05.149 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20299 2024-05-08T16:02:05.149 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20299/ceph-mon.a.asok 2024-05-08T16:02:05.149 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-05-08T16:02:05.150 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.20299/ceph-mon.a.asok config get mon_host 2024-05-08T16:02:05.264 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:46: TEST_preload_warning: run_mgr td/test-erasure-code-plugins x 2024-05-08T16:02:05.264 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/test-erasure-code-plugins 2024-05-08T16:02:05.264 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-05-08T16:02:05.264 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-05-08T16:02:05.264 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-05-08T16:02:05.264 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/test-erasure-code-plugins/x 2024-05-08T16:02:05.264 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-05-08T16:02:06.181 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-05-08T16:02:06.181 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-08T16:02:06.181 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-08T16:02:06.182 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-08T16:02:06.182 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T16:02:06.182 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20299 2024-05-08T16:02:06.182 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.20299/$cluster-$name.asok' 2024-05-08T16:02:06.183 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-05-08T16:02:06.184 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-code-plugins/x '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.20299/$cluster-$name.asok' --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-05-08T16:02:06.217 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:47: TEST_preload_warning: get_asok_path mon.a 2024-05-08T16:02:06.217 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-05-08T16:02:06.217 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-05-08T16:02:06.217 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-08T16:02:06.217 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T16:02:06.217 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20299 2024-05-08T16:02:06.217 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20299/ceph-mon.a.asok 2024-05-08T16:02:06.217 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:47: TEST_preload_warning: CEPH_ARGS= 2024-05-08T16:02:06.217 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:47: TEST_preload_warning: ceph --admin-daemon /tmp/ceph-asok.20299/ceph-mon.a.asok log flush 2024-05-08T16:02:06.310 INFO:tasks.workunit.client.0.smithi172.stdout:{} 2024-05-08T16:02:06.323 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:48: TEST_preload_warning: run_osd td/test-erasure-code-plugins 0 --osd_erasure_code_plugins=jerasure_sse3 2024-05-08T16:02:06.323 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-code-plugins 2024-05-08T16:02:06.323 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-05-08T16:02:06.323 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-05-08T16:02:06.323 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-05-08T16:02:06.323 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-code-plugins/0 2024-05-08T16:02:06.323 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=2a3a20f9-46fd-4090-ba60-6d102331c696 --auth-supported=none --mon-host=127.0.0.1:17110 ' 2024-05-08T16:02:06.323 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-08T16:02:06.323 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-05-08T16:02:06.323 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-08T16:02:06.323 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-code-plugins/0' 2024-05-08T16:02:06.323 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code-plugins/0/journal' 2024-05-08T16:02:06.324 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-05-08T16:02:06.324 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-05-08T16:02:06.324 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-code-plugins' 2024-05-08T16:02:06.326 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-05-08T16:02:06.326 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-08T16:02:06.326 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-08T16:02:06.326 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-08T16:02:06.326 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T16:02:06.326 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20299 2024-05-08T16:02:06.326 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.20299/$cluster-$name.asok' 2024-05-08T16:02:06.326 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.20299/$cluster-$name.asok' 2024-05-08T16:02:06.326 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-05-08T16:02:06.326 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-05-08T16:02:06.326 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-05-08T16:02:06.326 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-code-plugins/$name.log' 2024-05-08T16:02:06.326 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-code-plugins/$name.pid' 2024-05-08T16:02:06.327 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-08T16:02:06.327 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-08T16:02:06.327 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-08T16:02:06.327 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-08T16:02:06.327 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-05-08T16:02:06.327 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_erasure_code_plugins=jerasure_sse3 2024-05-08T16:02:06.327 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-code-plugins/0 2024-05-08T16:02:06.328 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-05-08T16:02:06.328 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=50c24569-a95c-4b7b-85b3-76df467643c9 2024-05-08T16:02:06.328 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 50c24569-a95c-4b7b-85b3-76df467643c9' 2024-05-08T16:02:06.328 INFO:tasks.workunit.client.0.smithi172.stdout:add osd0 50c24569-a95c-4b7b-85b3-76df467643c9 2024-05-08T16:02:06.328 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-05-08T16:02:06.348 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQD+oTtmkwojFRAAG8IwXc8aPBxhMx5svrN2hg== 2024-05-08T16:02:06.348 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQD+oTtmkwojFRAAG8IwXc8aPBxhMx5svrN2hg=="}' 2024-05-08T16:02:06.348 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 50c24569-a95c-4b7b-85b3-76df467643c9 -i td/test-erasure-code-plugins/0/new.json 2024-05-08T16:02:06.646 INFO:tasks.workunit.client.0.smithi172.stdout:0 2024-05-08T16:02:06.662 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-code-plugins/0/new.json 2024-05-08T16:02:06.664 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=2a3a20f9-46fd-4090-ba60-6d102331c696 --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.20299/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_erasure_code_plugins=jerasure_sse3 --mkfs --key AQD+oTtmkwojFRAAG8IwXc8aPBxhMx5svrN2hg== --osd-uuid 50c24569-a95c-4b7b-85b3-76df467643c9 2024-05-08T16:02:06.696 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:02:06.703+0000 7f0a606207c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:02:06.731 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:02:06.739+0000 7f0a606207c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:02:06.733 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:02:06.739+0000 7f0a606207c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:02:06.733 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:02:06.739+0000 7f0a606207c0 -1 bluestore(td/test-erasure-code-plugins/0/block) _read_bdev_label failed to open td/test-erasure-code-plugins/0/block: (2) No such file or directory 2024-05-08T16:02:06.733 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:02:06.739+0000 7f0a606207c0 -1 bluestore(td/test-erasure-code-plugins/0/block) _read_bdev_label failed to open td/test-erasure-code-plugins/0/block: (2) No such file or directory 2024-05-08T16:02:06.733 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:02:06.739+0000 7f0a606207c0 -1 bluestore(td/test-erasure-code-plugins/0/block) _read_bdev_label failed to open td/test-erasure-code-plugins/0/block: (2) No such file or directory 2024-05-08T16:02:06.771 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:02:06.779+0000 7f0a606207c0 -1 bluestore(td/test-erasure-code-plugins/0) _read_fsid unparsable uuid 2024-05-08T16:02:12.961 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-code-plugins/0/keyring 2024-05-08T16:02:12.961 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-05-08T16:02:12.963 INFO:tasks.workunit.client.0.smithi172.stdout:adding osd0 key to auth repository 2024-05-08T16:02:12.963 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-05-08T16:02:12.963 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-code-plugins/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-05-08T16:02:13.508 INFO:tasks.workunit.client.0.smithi172.stdout:start osd.0 2024-05-08T16:02:13.509 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-05-08T16:02:13.509 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=2a3a20f9-46fd-4090-ba60-6d102331c696 --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.20299/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_erasure_code_plugins=jerasure_sse3 2024-05-08T16:02:13.509 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-05-08T16:02:13.509 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-05-08T16:02:13.510 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-05-08T16:02:13.539 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:02:13.543+0000 7f7633f0d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:02:13.571 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:02:13.575+0000 7f7633f0d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:02:13.572 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:02:13.579+0000 7f7633f0d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:02:13.939 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-05-08T16:02:13.939 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-08T16:02:13.939 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-05-08T16:02:13.939 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-08T16:02:13.940 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-08T16:02:13.940 INFO:tasks.workunit.client.0.smithi172.stdout:0 2024-05-08T16:02:13.940 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:02:13.941 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-08T16:02:13.941 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:02:13.941 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:02:14.141 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:02:14.147+0000 7f7633f0d7c0 -1 Falling back to public interface 2024-05-08T16:02:14.350 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:02:15.352 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:02:15.352 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:02:15.352 INFO:tasks.workunit.client.0.smithi172.stdout:1 2024-05-08T16:02:15.353 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-08T16:02:15.353 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:02:15.353 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:02:15.746 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:02:16.678 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:02:16.683+0000 7f7633f0d7c0 -1 osd.0 0 log_to_monitors true 2024-05-08T16:02:16.748 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:02:16.748 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:02:16.748 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-08T16:02:16.748 INFO:tasks.workunit.client.0.smithi172.stdout:2 2024-05-08T16:02:16.749 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:02:16.749 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:02:17.246 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:02:18.248 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:02:18.248 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:02:18.248 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-05-08T16:02:18.249 INFO:tasks.workunit.client.0.smithi172.stdout:3 2024-05-08T16:02:18.249 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:02:18.249 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:02:18.719 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:02:19.721 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:02:19.721 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:02:19.721 INFO:tasks.workunit.client.0.smithi172.stdout:4 2024-05-08T16:02:19.722 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-05-08T16:02:19.722 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:02:19.722 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:02:20.172 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:02:21.175 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:02:21.175 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:02:21.175 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-05-08T16:02:21.175 INFO:tasks.workunit.client.0.smithi172.stdout:5 2024-05-08T16:02:21.175 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:02:21.176 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:02:21.730 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:02:22.732 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:02:22.733 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:02:22.733 INFO:tasks.workunit.client.0.smithi172.stdout:6 2024-05-08T16:02:22.733 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-05-08T16:02:22.734 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:02:22.734 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:02:23.176 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:02:24.178 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:02:24.178 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:02:24.178 INFO:tasks.workunit.client.0.smithi172.stdout:7 2024-05-08T16:02:24.179 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-05-08T16:02:24.179 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:02:24.179 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:02:24.723 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:02:25.725 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:02:25.725 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:02:25.725 INFO:tasks.workunit.client.0.smithi172.stdout:8 2024-05-08T16:02:25.726 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-05-08T16:02:25.726 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:02:25.726 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:02:26.195 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:02:27.197 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:02:27.198 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:02:27.198 INFO:tasks.workunit.client.0.smithi172.stdout:9 2024-05-08T16:02:27.198 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-05-08T16:02:27.198 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:02:27.198 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:02:27.777 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:02:28.779 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:02:28.779 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:02:28.780 INFO:tasks.workunit.client.0.smithi172.stdout:10 2024-05-08T16:02:28.780 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-05-08T16:02:28.780 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:02:28.780 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:02:29.177 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:02:30.179 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:02:30.180 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:02:30.180 INFO:tasks.workunit.client.0.smithi172.stdout:11 2024-05-08T16:02:30.180 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-05-08T16:02:30.181 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:02:30.181 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:02:31.467 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:02:32.469 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:02:32.469 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:02:32.469 INFO:tasks.workunit.client.0.smithi172.stdout:12 2024-05-08T16:02:32.470 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-05-08T16:02:32.470 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:02:32.470 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:02:33.296 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:02:34.298 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:02:34.298 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:02:34.299 INFO:tasks.workunit.client.0.smithi172.stdout:13 2024-05-08T16:02:34.299 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-05-08T16:02:34.299 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:02:34.299 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:02:35.051 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:02:36.052 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:02:36.053 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:02:36.053 INFO:tasks.workunit.client.0.smithi172.stdout:14 2024-05-08T16:02:36.053 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-05-08T16:02:36.053 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:02:36.053 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:02:36.926 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:02:37.928 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:02:37.928 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:02:37.929 INFO:tasks.workunit.client.0.smithi172.stdout:15 2024-05-08T16:02:37.929 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-05-08T16:02:37.929 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:02:37.929 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:02:38.681 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:02:39.686 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:02:39.686 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:02:39.686 INFO:tasks.workunit.client.0.smithi172.stdout:16 2024-05-08T16:02:39.687 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-05-08T16:02:39.687 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:02:39.688 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:02:40.493 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:02:41.495 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:02:41.495 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:02:41.496 INFO:tasks.workunit.client.0.smithi172.stdout:17 2024-05-08T16:02:41.497 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-05-08T16:02:41.497 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:02:41.497 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:02:41.941 INFO:tasks.workunit.client.0.smithi172.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3676947297,v1:127.0.0.1:6803/3676947297] [v2:127.0.0.1:6804/3676947297,v1:127.0.0.1:6805/3676947297] exists,up 50c24569-a95c-4b7b-85b3-76df467643c9 2024-05-08T16:02:41.942 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-08T16:02:41.942 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-08T16:02:41.942 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-08T16:02:41.943 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:49: TEST_preload_warning: get_asok_path osd.0 2024-05-08T16:02:41.943 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-05-08T16:02:41.943 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-05-08T16:02:41.943 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-08T16:02:41.944 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T16:02:41.944 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20299 2024-05-08T16:02:41.944 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20299/ceph-osd.0.asok 2024-05-08T16:02:41.944 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:49: TEST_preload_warning: CEPH_ARGS= 2024-05-08T16:02:41.944 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:49: TEST_preload_warning: ceph --admin-daemon /tmp/ceph-asok.20299/ceph-osd.0.asok log flush 2024-05-08T16:02:42.054 INFO:tasks.workunit.client.0.smithi172.stdout:{} 2024-05-08T16:02:42.067 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:50: TEST_preload_warning: grep 'WARNING: osd_erasure_code_plugins contains plugin jerasure_sse3' td/test-erasure-code-plugins/mon.a.log 2024-05-08T16:02:42.072 INFO:tasks.workunit.client.0.smithi172.stdout:2024-05-08T16:02:04.875+0000 7f0ea3bccc80 0 WARNING: osd_erasure_code_plugins contains plugin jerasure_sse3 that is now deprecated. Please modify the value for osd_erasure_code_plugins to use jerasure instead. 2024-05-08T16:02:42.072 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:51: TEST_preload_warning: grep 'WARNING: osd_erasure_code_plugins contains plugin jerasure_sse3' td/test-erasure-code-plugins/osd.0.log 2024-05-08T16:02:42.075 INFO:tasks.workunit.client.0.smithi172.stdout:2024-05-08T16:02:14.147+0000 7f7633f0d7c0 0 WARNING: osd_erasure_code_plugins contains plugin jerasure_sse3 that is now deprecated. Please modify the value for osd_erasure_code_plugins to use jerasure instead. 2024-05-08T16:02:42.076 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:52: TEST_preload_warning: teardown td/test-erasure-code-plugins 2024-05-08T16:02:42.076 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-code-plugins 2024-05-08T16:02:42.076 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-05-08T16:02:42.076 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-code-plugins KILL 2024-05-08T16:02:42.077 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-08T16:02:42.077 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-08T16:02:42.077 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-08T16:02:42.077 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-08T16:02:42.077 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-08T16:02:42.388 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-08T16:02:42.388 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-05-08T16:02:42.390 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-05-08T16:02:42.391 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-05-08T16:02:42.392 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-05-08T16:02:42.392 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-05-08T16:02:42.393 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-05-08T16:02:42.394 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-08T16:02:42.394 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-05-08T16:02:42.395 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-05-08T16:02:42.395 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-08T16:02:42.396 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-05-08T16:02:42.398 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-05-08T16:02:42.398 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-code-plugins 2024-05-08T16:02:42.405 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-05-08T16:02:42.406 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T16:02:42.406 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20299 2024-05-08T16:02:42.406 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.20299 2024-05-08T16:02:42.408 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-05-08T16:02:42.408 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-05-08T16:02:42.408 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:43: TEST_preload_warning: for plugin in ${legacy_jerasure_plugins[*]} ${legacy_shec_plugins[*]} 2024-05-08T16:02:42.408 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:44: TEST_preload_warning: setup td/test-erasure-code-plugins 2024-05-08T16:02:42.408 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/test-erasure-code-plugins 2024-05-08T16:02:42.408 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/test-erasure-code-plugins 2024-05-08T16:02:42.409 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-code-plugins 2024-05-08T16:02:42.409 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-05-08T16:02:42.409 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-code-plugins KILL 2024-05-08T16:02:42.409 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-08T16:02:42.409 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-08T16:02:42.409 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-08T16:02:42.409 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-08T16:02:42.409 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-08T16:02:42.412 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-08T16:02:42.412 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-05-08T16:02:42.413 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-05-08T16:02:42.413 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-05-08T16:02:42.414 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-05-08T16:02:42.414 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-05-08T16:02:42.415 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-05-08T16:02:42.416 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-08T16:02:42.416 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-05-08T16:02:42.416 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-05-08T16:02:42.417 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-08T16:02:42.417 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-05-08T16:02:42.419 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-05-08T16:02:42.419 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-code-plugins 2024-05-08T16:02:42.420 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-05-08T16:02:42.420 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T16:02:42.420 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20299 2024-05-08T16:02:42.421 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.20299 2024-05-08T16:02:42.421 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-05-08T16:02:42.422 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-05-08T16:02:42.422 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/test-erasure-code-plugins 2024-05-08T16:02:42.423 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-05-08T16:02:42.423 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T16:02:42.423 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20299 2024-05-08T16:02:42.425 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.20299 2024-05-08T16:02:42.427 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-05-08T16:02:42.427 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-05-08T16:02:42.428 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-05-08T16:02:42.428 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/test-erasure-code-plugins 1' TERM HUP INT 2024-05-08T16:02:42.428 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:45: TEST_preload_warning: run_mon td/test-erasure-code-plugins a --osd_erasure_code_plugins=jerasure_sse4 2024-05-08T16:02:42.428 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/test-erasure-code-plugins 2024-05-08T16:02:42.428 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-05-08T16:02:42.428 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-05-08T16:02:42.428 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-05-08T16:02:42.428 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/test-erasure-code-plugins/a 2024-05-08T16:02:42.428 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-code-plugins/a --run-dir=td/test-erasure-code-plugins --osd_erasure_code_plugins=jerasure_sse4 2024-05-08T16:02:42.824 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-05-08T16:02:42.824 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-08T16:02:42.824 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-08T16:02:42.825 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-08T16:02:42.825 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T16:02:42.825 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20299 2024-05-08T16:02:42.825 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.20299/$cluster-$name.asok' 2024-05-08T16:02:42.825 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-code-plugins/a '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.20299/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-code-plugins/log --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_erasure_code_plugins=jerasure_sse4 2024-05-08T16:02:43.049 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-05-08T16:02:43.050 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-05-08T16:02:43.050 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-05-08T16:02:43.050 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-05-08T16:02:43.050 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-05-08T16:02:43.051 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-05-08T16:02:43.051 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-05-08T16:02:43.051 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-05-08T16:02:43.051 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-05-08T16:02:43.052 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-08T16:02:43.052 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T16:02:43.052 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20299 2024-05-08T16:02:43.052 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20299/ceph-mon.a.asok 2024-05-08T16:02:43.052 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-05-08T16:02:43.052 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.20299/ceph-mon.a.asok config get fsid 2024-05-08T16:02:43.168 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-05-08T16:02:43.168 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-05-08T16:02:43.168 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-05-08T16:02:43.168 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-05-08T16:02:43.169 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-05-08T16:02:43.170 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-05-08T16:02:43.170 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-05-08T16:02:43.170 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-05-08T16:02:43.170 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-08T16:02:43.170 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T16:02:43.170 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20299 2024-05-08T16:02:43.170 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20299/ceph-mon.a.asok 2024-05-08T16:02:43.170 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-05-08T16:02:43.170 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.20299/ceph-mon.a.asok config get mon_host 2024-05-08T16:02:43.285 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:46: TEST_preload_warning: run_mgr td/test-erasure-code-plugins x 2024-05-08T16:02:43.285 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/test-erasure-code-plugins 2024-05-08T16:02:43.285 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-05-08T16:02:43.285 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-05-08T16:02:43.285 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-05-08T16:02:43.285 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/test-erasure-code-plugins/x 2024-05-08T16:02:43.285 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-05-08T16:02:44.205 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-05-08T16:02:44.205 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-08T16:02:44.205 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-08T16:02:44.205 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-08T16:02:44.205 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T16:02:44.205 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20299 2024-05-08T16:02:44.206 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.20299/$cluster-$name.asok' 2024-05-08T16:02:44.206 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-05-08T16:02:44.207 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-code-plugins/x '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.20299/$cluster-$name.asok' --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-05-08T16:02:44.242 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:47: TEST_preload_warning: get_asok_path mon.a 2024-05-08T16:02:44.242 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-05-08T16:02:44.243 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-05-08T16:02:44.243 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-08T16:02:44.243 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T16:02:44.243 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20299 2024-05-08T16:02:44.243 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20299/ceph-mon.a.asok 2024-05-08T16:02:44.243 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:47: TEST_preload_warning: CEPH_ARGS= 2024-05-08T16:02:44.243 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:47: TEST_preload_warning: ceph --admin-daemon /tmp/ceph-asok.20299/ceph-mon.a.asok log flush 2024-05-08T16:02:44.341 INFO:tasks.workunit.client.0.smithi172.stdout:{} 2024-05-08T16:02:44.356 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:48: TEST_preload_warning: run_osd td/test-erasure-code-plugins 0 --osd_erasure_code_plugins=jerasure_sse4 2024-05-08T16:02:44.356 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-code-plugins 2024-05-08T16:02:44.356 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-05-08T16:02:44.356 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-05-08T16:02:44.356 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-05-08T16:02:44.356 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-code-plugins/0 2024-05-08T16:02:44.356 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=2a3a20f9-46fd-4090-ba60-6d102331c696 --auth-supported=none --mon-host=127.0.0.1:17110 ' 2024-05-08T16:02:44.356 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-08T16:02:44.356 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-05-08T16:02:44.357 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-08T16:02:44.357 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-code-plugins/0' 2024-05-08T16:02:44.357 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code-plugins/0/journal' 2024-05-08T16:02:44.357 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-05-08T16:02:44.357 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-05-08T16:02:44.357 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-code-plugins' 2024-05-08T16:02:44.359 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-05-08T16:02:44.359 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-08T16:02:44.359 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-08T16:02:44.359 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-08T16:02:44.359 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T16:02:44.359 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20299 2024-05-08T16:02:44.359 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.20299/$cluster-$name.asok' 2024-05-08T16:02:44.359 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.20299/$cluster-$name.asok' 2024-05-08T16:02:44.359 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-05-08T16:02:44.359 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-05-08T16:02:44.359 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-05-08T16:02:44.360 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-code-plugins/$name.log' 2024-05-08T16:02:44.360 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-code-plugins/$name.pid' 2024-05-08T16:02:44.360 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-08T16:02:44.360 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-08T16:02:44.360 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-08T16:02:44.360 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-08T16:02:44.360 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-05-08T16:02:44.360 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_erasure_code_plugins=jerasure_sse4 2024-05-08T16:02:44.360 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-code-plugins/0 2024-05-08T16:02:44.360 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-05-08T16:02:44.361 INFO:tasks.workunit.client.0.smithi172.stdout:add osd0 1ed81a3a-0e91-4770-b235-01bff727ff21 2024-05-08T16:02:44.361 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=1ed81a3a-0e91-4770-b235-01bff727ff21 2024-05-08T16:02:44.361 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 1ed81a3a-0e91-4770-b235-01bff727ff21' 2024-05-08T16:02:44.361 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-05-08T16:02:44.380 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAkojtmpqYTFxAAozDwyaT7jx+ItRkO6lSH8w== 2024-05-08T16:02:44.380 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAkojtmpqYTFxAAozDwyaT7jx+ItRkO6lSH8w=="}' 2024-05-08T16:02:44.380 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 1ed81a3a-0e91-4770-b235-01bff727ff21 -i td/test-erasure-code-plugins/0/new.json 2024-05-08T16:02:44.713 INFO:tasks.workunit.client.0.smithi172.stdout:0 2024-05-08T16:02:44.729 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-code-plugins/0/new.json 2024-05-08T16:02:44.730 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=2a3a20f9-46fd-4090-ba60-6d102331c696 --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.20299/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_erasure_code_plugins=jerasure_sse4 --mkfs --key AQAkojtmpqYTFxAAozDwyaT7jx+ItRkO6lSH8w== --osd-uuid 1ed81a3a-0e91-4770-b235-01bff727ff21 2024-05-08T16:02:44.761 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:02:44.766+0000 7f50f145e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:02:44.781 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:02:44.786+0000 7f50f145e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:02:44.782 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:02:44.786+0000 7f50f145e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:02:44.783 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:02:44.786+0000 7f50f145e7c0 -1 bluestore(td/test-erasure-code-plugins/0/block) _read_bdev_label failed to open td/test-erasure-code-plugins/0/block: (2) No such file or directory 2024-05-08T16:02:44.783 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:02:44.786+0000 7f50f145e7c0 -1 bluestore(td/test-erasure-code-plugins/0/block) _read_bdev_label failed to open td/test-erasure-code-plugins/0/block: (2) No such file or directory 2024-05-08T16:02:44.783 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:02:44.786+0000 7f50f145e7c0 -1 bluestore(td/test-erasure-code-plugins/0/block) _read_bdev_label failed to open td/test-erasure-code-plugins/0/block: (2) No such file or directory 2024-05-08T16:02:44.813 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:02:44.818+0000 7f50f145e7c0 -1 bluestore(td/test-erasure-code-plugins/0) _read_fsid unparsable uuid 2024-05-08T16:02:51.032 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-code-plugins/0/keyring 2024-05-08T16:02:51.032 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-05-08T16:02:51.034 INFO:tasks.workunit.client.0.smithi172.stdout:adding osd0 key to auth repository 2024-05-08T16:02:51.034 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-05-08T16:02:51.034 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-code-plugins/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-05-08T16:02:51.573 INFO:tasks.workunit.client.0.smithi172.stdout:start osd.0 2024-05-08T16:02:51.574 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-05-08T16:02:51.574 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=2a3a20f9-46fd-4090-ba60-6d102331c696 --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.20299/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_erasure_code_plugins=jerasure_sse4 2024-05-08T16:02:51.574 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-05-08T16:02:51.574 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-05-08T16:02:51.574 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-05-08T16:02:51.604 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:02:51.610+0000 7ff0acba97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:02:51.606 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:02:51.610+0000 7ff0acba97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:02:51.607 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:02:51.614+0000 7ff0acba97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:02:51.982 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-05-08T16:02:51.982 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-08T16:02:51.982 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-05-08T16:02:51.982 INFO:tasks.workunit.client.0.smithi172.stdout:0 2024-05-08T16:02:51.983 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-08T16:02:51.983 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-08T16:02:51.983 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:02:51.983 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-08T16:02:51.983 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:02:51.983 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:02:52.176 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:02:52.182+0000 7ff0acba97c0 -1 Falling back to public interface 2024-05-08T16:02:52.404 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:02:53.406 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:02:53.406 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:02:53.406 INFO:tasks.workunit.client.0.smithi172.stdout:1 2024-05-08T16:02:53.407 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-08T16:02:53.407 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:02:53.407 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:02:53.807 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:02:54.796 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:02:54.802+0000 7ff0acba97c0 -1 osd.0 0 log_to_monitors true 2024-05-08T16:02:54.808 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:02:54.808 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:02:54.808 INFO:tasks.workunit.client.0.smithi172.stdout:2 2024-05-08T16:02:54.809 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-08T16:02:54.809 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:02:54.809 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:02:55.546 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:02:56.547 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:02:56.547 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:02:56.548 INFO:tasks.workunit.client.0.smithi172.stdout:3 2024-05-08T16:02:56.548 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-05-08T16:02:56.548 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:02:56.548 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:02:56.984 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:02:57.986 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:02:57.986 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:02:57.986 INFO:tasks.workunit.client.0.smithi172.stdout:4 2024-05-08T16:02:57.987 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-05-08T16:02:57.987 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:02:57.987 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:02:58.430 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:02:59.432 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:02:59.432 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:02:59.433 INFO:tasks.workunit.client.0.smithi172.stdout:5 2024-05-08T16:02:59.433 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-05-08T16:02:59.433 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:02:59.433 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:02:59.967 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:03:00.968 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:03:00.969 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:03:00.969 INFO:tasks.workunit.client.0.smithi172.stdout:6 2024-05-08T16:03:00.969 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-05-08T16:03:00.969 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:03:00.969 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:03:01.397 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:03:02.399 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:03:02.399 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:03:02.399 INFO:tasks.workunit.client.0.smithi172.stdout:7 2024-05-08T16:03:02.400 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-05-08T16:03:02.400 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:03:02.400 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:03:02.933 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:03:03.935 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:03:03.935 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:03:03.936 INFO:tasks.workunit.client.0.smithi172.stdout:8 2024-05-08T16:03:03.937 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-05-08T16:03:03.937 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:03:03.937 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:03:04.371 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:03:05.372 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:03:05.373 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:03:05.373 INFO:tasks.workunit.client.0.smithi172.stdout:9 2024-05-08T16:03:05.373 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-05-08T16:03:05.374 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:03:05.374 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:03:05.941 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:03:06.943 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:03:06.943 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:03:06.943 INFO:tasks.workunit.client.0.smithi172.stdout:10 2024-05-08T16:03:06.944 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-05-08T16:03:06.944 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:03:06.944 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:03:07.345 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:03:08.347 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:03:08.347 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:03:08.347 INFO:tasks.workunit.client.0.smithi172.stdout:11 2024-05-08T16:03:08.348 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-05-08T16:03:08.348 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:03:08.348 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:03:09.190 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:03:10.192 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:03:10.192 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:03:10.193 INFO:tasks.workunit.client.0.smithi172.stdout:12 2024-05-08T16:03:10.193 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-05-08T16:03:10.193 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:03:10.193 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:03:11.084 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:03:12.086 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:03:12.086 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:03:12.086 INFO:tasks.workunit.client.0.smithi172.stdout:13 2024-05-08T16:03:12.087 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-05-08T16:03:12.087 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:03:12.087 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:03:12.919 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:03:13.921 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:03:13.921 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:03:13.922 INFO:tasks.workunit.client.0.smithi172.stdout:14 2024-05-08T16:03:13.923 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-05-08T16:03:13.923 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:03:13.923 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:03:14.747 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:03:15.749 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:03:15.749 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:03:15.750 INFO:tasks.workunit.client.0.smithi172.stdout:15 2024-05-08T16:03:15.750 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-05-08T16:03:15.750 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:03:15.750 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:03:16.614 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:03:17.616 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:03:17.616 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:03:17.617 INFO:tasks.workunit.client.0.smithi172.stdout:16 2024-05-08T16:03:17.617 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-05-08T16:03:17.617 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:03:17.617 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:03:18.825 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:03:19.827 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:03:19.827 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:03:19.827 INFO:tasks.workunit.client.0.smithi172.stdout:17 2024-05-08T16:03:19.828 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-05-08T16:03:19.828 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:03:19.828 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:03:20.314 INFO:tasks.workunit.client.0.smithi172.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3445137266,v1:127.0.0.1:6803/3445137266] [v2:127.0.0.1:6804/3445137266,v1:127.0.0.1:6805/3445137266] exists,up 1ed81a3a-0e91-4770-b235-01bff727ff21 2024-05-08T16:03:20.314 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-08T16:03:20.314 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-08T16:03:20.314 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-08T16:03:20.315 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:49: TEST_preload_warning: get_asok_path osd.0 2024-05-08T16:03:20.315 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-05-08T16:03:20.315 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-05-08T16:03:20.316 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-08T16:03:20.316 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T16:03:20.316 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20299 2024-05-08T16:03:20.316 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20299/ceph-osd.0.asok 2024-05-08T16:03:20.316 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:49: TEST_preload_warning: CEPH_ARGS= 2024-05-08T16:03:20.317 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:49: TEST_preload_warning: ceph --admin-daemon /tmp/ceph-asok.20299/ceph-osd.0.asok log flush 2024-05-08T16:03:20.424 INFO:tasks.workunit.client.0.smithi172.stdout:{} 2024-05-08T16:03:20.436 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:50: TEST_preload_warning: grep 'WARNING: osd_erasure_code_plugins contains plugin jerasure_sse4' td/test-erasure-code-plugins/mon.a.log 2024-05-08T16:03:20.440 INFO:tasks.workunit.client.0.smithi172.stdout:2024-05-08T16:02:42.866+0000 7f5665c54c80 0 WARNING: osd_erasure_code_plugins contains plugin jerasure_sse4 that is now deprecated. Please modify the value for osd_erasure_code_plugins to use jerasure instead. 2024-05-08T16:03:20.441 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:51: TEST_preload_warning: grep 'WARNING: osd_erasure_code_plugins contains plugin jerasure_sse4' td/test-erasure-code-plugins/osd.0.log 2024-05-08T16:03:20.442 INFO:tasks.workunit.client.0.smithi172.stdout:2024-05-08T16:02:52.182+0000 7ff0acba97c0 0 WARNING: osd_erasure_code_plugins contains plugin jerasure_sse4 that is now deprecated. Please modify the value for osd_erasure_code_plugins to use jerasure instead. 2024-05-08T16:03:20.443 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:52: TEST_preload_warning: teardown td/test-erasure-code-plugins 2024-05-08T16:03:20.443 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-code-plugins 2024-05-08T16:03:20.444 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-05-08T16:03:20.444 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-code-plugins KILL 2024-05-08T16:03:20.444 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-08T16:03:20.444 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-08T16:03:20.444 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-08T16:03:20.444 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-08T16:03:20.444 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-08T16:03:20.553 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-08T16:03:20.553 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-05-08T16:03:20.554 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-05-08T16:03:20.555 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-05-08T16:03:20.555 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-05-08T16:03:20.556 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-05-08T16:03:20.556 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-05-08T16:03:20.557 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-08T16:03:20.557 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-05-08T16:03:20.557 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-05-08T16:03:20.558 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-08T16:03:20.558 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-05-08T16:03:20.559 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-05-08T16:03:20.559 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-code-plugins 2024-05-08T16:03:20.565 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-05-08T16:03:20.565 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T16:03:20.565 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20299 2024-05-08T16:03:20.565 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.20299 2024-05-08T16:03:20.569 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-05-08T16:03:20.569 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-05-08T16:03:20.569 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:43: TEST_preload_warning: for plugin in ${legacy_jerasure_plugins[*]} ${legacy_shec_plugins[*]} 2024-05-08T16:03:20.569 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:44: TEST_preload_warning: setup td/test-erasure-code-plugins 2024-05-08T16:03:20.569 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/test-erasure-code-plugins 2024-05-08T16:03:20.569 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/test-erasure-code-plugins 2024-05-08T16:03:20.569 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-code-plugins 2024-05-08T16:03:20.569 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-05-08T16:03:20.569 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-code-plugins KILL 2024-05-08T16:03:20.569 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-08T16:03:20.569 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-08T16:03:20.569 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-08T16:03:20.570 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-08T16:03:20.570 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-08T16:03:20.572 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-08T16:03:20.572 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-05-08T16:03:20.573 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-05-08T16:03:20.573 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-05-08T16:03:20.574 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-05-08T16:03:20.574 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-05-08T16:03:20.574 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-05-08T16:03:20.575 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-08T16:03:20.575 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-05-08T16:03:20.576 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-05-08T16:03:20.576 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-08T16:03:20.577 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-05-08T16:03:20.578 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-05-08T16:03:20.578 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-code-plugins 2024-05-08T16:03:20.579 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-05-08T16:03:20.579 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T16:03:20.579 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20299 2024-05-08T16:03:20.579 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.20299 2024-05-08T16:03:20.580 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-05-08T16:03:20.580 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-05-08T16:03:20.580 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/test-erasure-code-plugins 2024-05-08T16:03:20.581 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-05-08T16:03:20.581 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T16:03:20.581 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20299 2024-05-08T16:03:20.582 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.20299 2024-05-08T16:03:20.584 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-05-08T16:03:20.584 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-05-08T16:03:20.584 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-05-08T16:03:20.584 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/test-erasure-code-plugins 1' TERM HUP INT 2024-05-08T16:03:20.584 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:45: TEST_preload_warning: run_mon td/test-erasure-code-plugins a --osd_erasure_code_plugins=shec_generic 2024-05-08T16:03:20.584 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/test-erasure-code-plugins 2024-05-08T16:03:20.584 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-05-08T16:03:20.584 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-05-08T16:03:20.584 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-05-08T16:03:20.584 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/test-erasure-code-plugins/a 2024-05-08T16:03:20.584 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-code-plugins/a --run-dir=td/test-erasure-code-plugins --osd_erasure_code_plugins=shec_generic 2024-05-08T16:03:21.035 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-05-08T16:03:21.035 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-08T16:03:21.035 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-08T16:03:21.036 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-08T16:03:21.036 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T16:03:21.036 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20299 2024-05-08T16:03:21.036 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.20299/$cluster-$name.asok' 2024-05-08T16:03:21.036 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-code-plugins/a '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.20299/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-code-plugins/log --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_erasure_code_plugins=shec_generic 2024-05-08T16:03:21.236 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-05-08T16:03:21.237 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-05-08T16:03:21.237 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-05-08T16:03:21.237 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-05-08T16:03:21.237 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-05-08T16:03:21.238 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-05-08T16:03:21.238 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-05-08T16:03:21.238 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-05-08T16:03:21.239 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-05-08T16:03:21.239 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-08T16:03:21.239 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T16:03:21.239 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20299 2024-05-08T16:03:21.239 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20299/ceph-mon.a.asok 2024-05-08T16:03:21.239 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-05-08T16:03:21.240 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.20299/ceph-mon.a.asok config get fsid 2024-05-08T16:03:21.356 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-05-08T16:03:21.356 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-05-08T16:03:21.356 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-05-08T16:03:21.356 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-05-08T16:03:21.357 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-05-08T16:03:21.357 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-05-08T16:03:21.357 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-05-08T16:03:21.357 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-05-08T16:03:21.357 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-08T16:03:21.357 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T16:03:21.357 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20299 2024-05-08T16:03:21.357 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20299/ceph-mon.a.asok 2024-05-08T16:03:21.358 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-05-08T16:03:21.358 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.20299/ceph-mon.a.asok config get mon_host 2024-05-08T16:03:21.468 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:46: TEST_preload_warning: run_mgr td/test-erasure-code-plugins x 2024-05-08T16:03:21.469 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/test-erasure-code-plugins 2024-05-08T16:03:21.469 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-05-08T16:03:21.469 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-05-08T16:03:21.469 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-05-08T16:03:21.469 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/test-erasure-code-plugins/x 2024-05-08T16:03:21.469 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-05-08T16:03:22.385 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-05-08T16:03:22.385 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-08T16:03:22.385 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-08T16:03:22.386 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-08T16:03:22.386 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T16:03:22.386 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20299 2024-05-08T16:03:22.386 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.20299/$cluster-$name.asok' 2024-05-08T16:03:22.387 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-05-08T16:03:22.388 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-code-plugins/x '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.20299/$cluster-$name.asok' --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-05-08T16:03:22.421 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:47: TEST_preload_warning: get_asok_path mon.a 2024-05-08T16:03:22.421 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-05-08T16:03:22.421 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-05-08T16:03:22.421 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-08T16:03:22.421 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T16:03:22.421 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20299 2024-05-08T16:03:22.421 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20299/ceph-mon.a.asok 2024-05-08T16:03:22.422 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:47: TEST_preload_warning: CEPH_ARGS= 2024-05-08T16:03:22.422 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:47: TEST_preload_warning: ceph --admin-daemon /tmp/ceph-asok.20299/ceph-mon.a.asok log flush 2024-05-08T16:03:22.516 INFO:tasks.workunit.client.0.smithi172.stdout:{} 2024-05-08T16:03:22.531 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:48: TEST_preload_warning: run_osd td/test-erasure-code-plugins 0 --osd_erasure_code_plugins=shec_generic 2024-05-08T16:03:22.531 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-code-plugins 2024-05-08T16:03:22.531 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-05-08T16:03:22.532 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-05-08T16:03:22.532 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-05-08T16:03:22.532 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-code-plugins/0 2024-05-08T16:03:22.532 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=2a3a20f9-46fd-4090-ba60-6d102331c696 --auth-supported=none --mon-host=127.0.0.1:17110 ' 2024-05-08T16:03:22.532 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-08T16:03:22.532 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-05-08T16:03:22.532 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-08T16:03:22.532 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-code-plugins/0' 2024-05-08T16:03:22.532 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code-plugins/0/journal' 2024-05-08T16:03:22.532 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-05-08T16:03:22.532 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-05-08T16:03:22.532 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-code-plugins' 2024-05-08T16:03:22.532 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-05-08T16:03:22.532 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-08T16:03:22.533 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-08T16:03:22.533 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-08T16:03:22.533 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T16:03:22.533 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20299 2024-05-08T16:03:22.533 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.20299/$cluster-$name.asok' 2024-05-08T16:03:22.534 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.20299/$cluster-$name.asok' 2024-05-08T16:03:22.535 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-05-08T16:03:22.535 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-05-08T16:03:22.535 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-05-08T16:03:22.535 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-code-plugins/$name.log' 2024-05-08T16:03:22.535 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-code-plugins/$name.pid' 2024-05-08T16:03:22.535 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-08T16:03:22.535 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-08T16:03:22.535 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-08T16:03:22.535 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-08T16:03:22.535 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-05-08T16:03:22.535 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_erasure_code_plugins=shec_generic 2024-05-08T16:03:22.535 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-code-plugins/0 2024-05-08T16:03:22.535 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-05-08T16:03:22.535 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=34b658ed-fb86-40f3-9853-5f3904b5c9f9 2024-05-08T16:03:22.536 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 34b658ed-fb86-40f3-9853-5f3904b5c9f9' 2024-05-08T16:03:22.536 INFO:tasks.workunit.client.0.smithi172.stdout:add osd0 34b658ed-fb86-40f3-9853-5f3904b5c9f9 2024-05-08T16:03:22.536 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-05-08T16:03:22.550 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBKojtmymA7IRAAcjV9vOl+yO+A4xIED+RwWQ== 2024-05-08T16:03:22.550 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBKojtmymA7IRAAcjV9vOl+yO+A4xIED+RwWQ=="}' 2024-05-08T16:03:22.550 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 34b658ed-fb86-40f3-9853-5f3904b5c9f9 -i td/test-erasure-code-plugins/0/new.json 2024-05-08T16:03:22.841 INFO:tasks.workunit.client.0.smithi172.stdout:0 2024-05-08T16:03:22.856 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-code-plugins/0/new.json 2024-05-08T16:03:22.857 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=2a3a20f9-46fd-4090-ba60-6d102331c696 --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.20299/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_erasure_code_plugins=shec_generic --mkfs --key AQBKojtmymA7IRAAcjV9vOl+yO+A4xIED+RwWQ== --osd-uuid 34b658ed-fb86-40f3-9853-5f3904b5c9f9 2024-05-08T16:03:22.893 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:03:22.897+0000 7f54eddb57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:03:22.908 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:03:22.913+0000 7f54eddb57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:03:22.910 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:03:22.913+0000 7f54eddb57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:03:22.910 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:03:22.913+0000 7f54eddb57c0 -1 bluestore(td/test-erasure-code-plugins/0/block) _read_bdev_label failed to open td/test-erasure-code-plugins/0/block: (2) No such file or directory 2024-05-08T16:03:22.911 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:03:22.913+0000 7f54eddb57c0 -1 bluestore(td/test-erasure-code-plugins/0/block) _read_bdev_label failed to open td/test-erasure-code-plugins/0/block: (2) No such file or directory 2024-05-08T16:03:22.911 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:03:22.913+0000 7f54eddb57c0 -1 bluestore(td/test-erasure-code-plugins/0/block) _read_bdev_label failed to open td/test-erasure-code-plugins/0/block: (2) No such file or directory 2024-05-08T16:03:22.942 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:03:22.945+0000 7f54eddb57c0 -1 bluestore(td/test-erasure-code-plugins/0) _read_fsid unparsable uuid 2024-05-08T16:03:29.209 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-code-plugins/0/keyring 2024-05-08T16:03:29.209 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-05-08T16:03:29.210 INFO:tasks.workunit.client.0.smithi172.stdout:adding osd0 key to auth repository 2024-05-08T16:03:29.211 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-05-08T16:03:29.211 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-code-plugins/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-05-08T16:03:29.740 INFO:tasks.workunit.client.0.smithi172.stdout:start osd.0 2024-05-08T16:03:29.741 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-05-08T16:03:29.741 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=2a3a20f9-46fd-4090-ba60-6d102331c696 --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.20299/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_erasure_code_plugins=shec_generic 2024-05-08T16:03:29.741 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-05-08T16:03:29.741 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-05-08T16:03:29.741 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-05-08T16:03:29.770 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:03:29.773+0000 7fa5a04237c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:03:29.773 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:03:29.777+0000 7fa5a04237c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:03:29.774 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:03:29.777+0000 7fa5a04237c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:03:30.186 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-05-08T16:03:30.186 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-08T16:03:30.186 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-05-08T16:03:30.186 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-08T16:03:30.187 INFO:tasks.workunit.client.0.smithi172.stdout:0 2024-05-08T16:03:30.187 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-08T16:03:30.187 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:03:30.187 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-08T16:03:30.187 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:03:30.188 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:03:30.333 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:03:30.337+0000 7fa5a04237c0 -1 Falling back to public interface 2024-05-08T16:03:30.619 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:03:31.621 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:03:31.621 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:03:31.621 INFO:tasks.workunit.client.0.smithi172.stdout:1 2024-05-08T16:03:31.621 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-08T16:03:31.622 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:03:31.622 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:03:32.039 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:03:32.980 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:03:32.985+0000 7fa5a04237c0 -1 osd.0 0 log_to_monitors true 2024-05-08T16:03:33.040 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:03:33.040 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:03:33.041 INFO:tasks.workunit.client.0.smithi172.stdout:2 2024-05-08T16:03:33.041 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-08T16:03:33.041 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:03:33.041 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:03:33.721 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:03:34.723 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:03:34.723 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:03:34.723 INFO:tasks.workunit.client.0.smithi172.stdout:3 2024-05-08T16:03:34.724 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-05-08T16:03:34.724 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:03:34.724 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:03:35.127 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:03:36.130 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:03:36.130 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:03:36.130 INFO:tasks.workunit.client.0.smithi172.stdout:4 2024-05-08T16:03:36.131 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-05-08T16:03:36.131 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:03:36.131 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:03:36.572 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:03:37.574 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:03:37.574 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:03:37.574 INFO:tasks.workunit.client.0.smithi172.stdout:5 2024-05-08T16:03:37.575 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-05-08T16:03:37.575 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:03:37.575 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:03:37.972 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:03:38.973 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:03:38.973 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:03:38.974 INFO:tasks.workunit.client.0.smithi172.stdout:6 2024-05-08T16:03:38.974 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-05-08T16:03:38.974 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:03:38.975 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:03:39.401 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:03:40.402 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:03:40.403 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:03:40.403 INFO:tasks.workunit.client.0.smithi172.stdout:7 2024-05-08T16:03:40.404 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-05-08T16:03:40.404 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:03:40.404 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:03:40.966 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:03:41.967 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:03:41.967 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:03:41.968 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-05-08T16:03:41.968 INFO:tasks.workunit.client.0.smithi172.stdout:8 2024-05-08T16:03:41.968 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:03:41.968 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:03:42.383 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:03:43.384 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:03:43.384 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:03:43.385 INFO:tasks.workunit.client.0.smithi172.stdout:9 2024-05-08T16:03:43.385 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-05-08T16:03:43.385 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:03:43.385 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:03:43.887 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:03:44.889 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:03:44.890 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:03:44.890 INFO:tasks.workunit.client.0.smithi172.stdout:10 2024-05-08T16:03:44.890 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-05-08T16:03:44.891 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:03:44.891 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:03:45.320 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:03:46.321 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:03:46.322 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:03:46.322 INFO:tasks.workunit.client.0.smithi172.stdout:11 2024-05-08T16:03:46.323 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-05-08T16:03:46.323 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:03:46.323 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:03:46.966 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:03:47.967 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:03:47.967 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:03:47.967 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-05-08T16:03:47.968 INFO:tasks.workunit.client.0.smithi172.stdout:12 2024-05-08T16:03:47.968 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:03:47.968 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:03:48.813 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:03:49.815 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:03:49.815 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:03:49.815 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-05-08T16:03:49.816 INFO:tasks.workunit.client.0.smithi172.stdout:13 2024-05-08T16:03:49.816 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:03:49.816 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:03:50.647 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:03:51.649 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:03:51.649 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:03:51.650 INFO:tasks.workunit.client.0.smithi172.stdout:14 2024-05-08T16:03:51.650 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-05-08T16:03:51.650 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:03:51.650 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:03:52.517 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:03:53.519 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:03:53.519 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:03:53.519 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-05-08T16:03:53.520 INFO:tasks.workunit.client.0.smithi172.stdout:15 2024-05-08T16:03:53.520 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:03:53.520 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:03:54.186 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:03:55.187 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:03:55.188 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:03:55.188 INFO:tasks.workunit.client.0.smithi172.stdout:16 2024-05-08T16:03:55.188 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-05-08T16:03:55.189 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:03:55.189 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:03:55.959 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:03:56.961 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:03:56.961 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:03:56.961 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-05-08T16:03:56.961 INFO:tasks.workunit.client.0.smithi172.stdout:17 2024-05-08T16:03:56.962 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:03:56.962 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:03:57.504 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:03:58.506 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:03:58.506 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:03:58.507 INFO:tasks.workunit.client.0.smithi172.stdout:18 2024-05-08T16:03:58.507 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-05-08T16:03:58.507 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:03:58.507 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:03:58.990 INFO:tasks.workunit.client.0.smithi172.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3272277981,v1:127.0.0.1:6803/3272277981] [v2:127.0.0.1:6804/3272277981,v1:127.0.0.1:6805/3272277981] exists,up 34b658ed-fb86-40f3-9853-5f3904b5c9f9 2024-05-08T16:03:58.991 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-08T16:03:58.991 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-08T16:03:58.992 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-08T16:03:58.992 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:49: TEST_preload_warning: get_asok_path osd.0 2024-05-08T16:03:58.992 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-05-08T16:03:58.992 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-05-08T16:03:58.992 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-08T16:03:58.992 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T16:03:58.992 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20299 2024-05-08T16:03:58.992 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20299/ceph-osd.0.asok 2024-05-08T16:03:58.992 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:49: TEST_preload_warning: CEPH_ARGS= 2024-05-08T16:03:58.992 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:49: TEST_preload_warning: ceph --admin-daemon /tmp/ceph-asok.20299/ceph-osd.0.asok log flush 2024-05-08T16:03:59.104 INFO:tasks.workunit.client.0.smithi172.stdout:{} 2024-05-08T16:03:59.116 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:50: TEST_preload_warning: grep 'WARNING: osd_erasure_code_plugins contains plugin shec_generic' td/test-erasure-code-plugins/mon.a.log 2024-05-08T16:03:59.120 INFO:tasks.workunit.client.0.smithi172.stdout:2024-05-08T16:03:21.077+0000 7f7da2bb4c80 0 WARNING: osd_erasure_code_plugins contains plugin shec_generic that is now deprecated. Please modify the value for osd_erasure_code_plugins to use shec instead. 2024-05-08T16:03:59.121 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:51: TEST_preload_warning: grep 'WARNING: osd_erasure_code_plugins contains plugin shec_generic' td/test-erasure-code-plugins/osd.0.log 2024-05-08T16:03:59.124 INFO:tasks.workunit.client.0.smithi172.stdout:2024-05-08T16:03:30.337+0000 7fa5a04237c0 0 WARNING: osd_erasure_code_plugins contains plugin shec_generic that is now deprecated. Please modify the value for osd_erasure_code_plugins to use shec instead. 2024-05-08T16:03:59.125 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:52: TEST_preload_warning: teardown td/test-erasure-code-plugins 2024-05-08T16:03:59.125 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-code-plugins 2024-05-08T16:03:59.125 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-05-08T16:03:59.125 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-code-plugins KILL 2024-05-08T16:03:59.125 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-08T16:03:59.125 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-08T16:03:59.126 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-08T16:03:59.126 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-08T16:03:59.126 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-08T16:03:59.435 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-08T16:03:59.435 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-05-08T16:03:59.436 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-05-08T16:03:59.437 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-05-08T16:03:59.439 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-05-08T16:03:59.439 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-05-08T16:03:59.439 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-05-08T16:03:59.440 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-08T16:03:59.440 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-05-08T16:03:59.441 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-05-08T16:03:59.441 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-08T16:03:59.442 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-05-08T16:03:59.443 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-05-08T16:03:59.443 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-code-plugins 2024-05-08T16:03:59.452 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-05-08T16:03:59.452 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T16:03:59.452 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20299 2024-05-08T16:03:59.452 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.20299 2024-05-08T16:03:59.454 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-05-08T16:03:59.455 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-05-08T16:03:59.455 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:43: TEST_preload_warning: for plugin in ${legacy_jerasure_plugins[*]} ${legacy_shec_plugins[*]} 2024-05-08T16:03:59.455 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:44: TEST_preload_warning: setup td/test-erasure-code-plugins 2024-05-08T16:03:59.455 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/test-erasure-code-plugins 2024-05-08T16:03:59.455 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/test-erasure-code-plugins 2024-05-08T16:03:59.455 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-code-plugins 2024-05-08T16:03:59.455 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-05-08T16:03:59.455 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-code-plugins KILL 2024-05-08T16:03:59.455 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-08T16:03:59.455 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-08T16:03:59.455 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-08T16:03:59.455 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-08T16:03:59.455 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-08T16:03:59.457 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-08T16:03:59.457 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-05-08T16:03:59.458 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-05-08T16:03:59.459 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-05-08T16:03:59.460 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-05-08T16:03:59.460 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-05-08T16:03:59.460 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-05-08T16:03:59.461 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-08T16:03:59.461 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-05-08T16:03:59.461 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-05-08T16:03:59.462 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-08T16:03:59.462 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-05-08T16:03:59.464 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-05-08T16:03:59.464 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-code-plugins 2024-05-08T16:03:59.465 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-05-08T16:03:59.465 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T16:03:59.465 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20299 2024-05-08T16:03:59.465 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.20299 2024-05-08T16:03:59.467 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-05-08T16:03:59.467 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-05-08T16:03:59.467 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/test-erasure-code-plugins 2024-05-08T16:03:59.468 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-05-08T16:03:59.468 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T16:03:59.468 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20299 2024-05-08T16:03:59.469 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.20299 2024-05-08T16:03:59.471 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-05-08T16:03:59.471 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-05-08T16:03:59.471 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-05-08T16:03:59.471 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/test-erasure-code-plugins 1' TERM HUP INT 2024-05-08T16:03:59.471 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:45: TEST_preload_warning: run_mon td/test-erasure-code-plugins a --osd_erasure_code_plugins=shec_sse3 2024-05-08T16:03:59.471 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/test-erasure-code-plugins 2024-05-08T16:03:59.471 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-05-08T16:03:59.471 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-05-08T16:03:59.471 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-05-08T16:03:59.472 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/test-erasure-code-plugins/a 2024-05-08T16:03:59.472 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-code-plugins/a --run-dir=td/test-erasure-code-plugins --osd_erasure_code_plugins=shec_sse3 2024-05-08T16:03:59.873 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-05-08T16:03:59.873 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-08T16:03:59.873 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-08T16:03:59.874 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-08T16:03:59.874 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T16:03:59.874 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20299 2024-05-08T16:03:59.874 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.20299/$cluster-$name.asok' 2024-05-08T16:03:59.874 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-code-plugins/a '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.20299/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-code-plugins/log --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_erasure_code_plugins=shec_sse3 2024-05-08T16:04:00.092 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-05-08T16:04:00.093 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-05-08T16:04:00.093 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-05-08T16:04:00.093 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-05-08T16:04:00.093 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-05-08T16:04:00.094 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-05-08T16:04:00.095 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-05-08T16:04:00.095 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-05-08T16:04:00.095 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-05-08T16:04:00.095 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-08T16:04:00.095 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T16:04:00.095 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20299 2024-05-08T16:04:00.095 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20299/ceph-mon.a.asok 2024-05-08T16:04:00.095 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-05-08T16:04:00.095 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.20299/ceph-mon.a.asok config get fsid 2024-05-08T16:04:00.207 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-05-08T16:04:00.207 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-05-08T16:04:00.207 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-05-08T16:04:00.208 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-05-08T16:04:00.208 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-05-08T16:04:00.209 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-05-08T16:04:00.209 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-05-08T16:04:00.209 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-05-08T16:04:00.209 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-08T16:04:00.209 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T16:04:00.209 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20299 2024-05-08T16:04:00.209 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20299/ceph-mon.a.asok 2024-05-08T16:04:00.210 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-05-08T16:04:00.210 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.20299/ceph-mon.a.asok config get mon_host 2024-05-08T16:04:00.321 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:46: TEST_preload_warning: run_mgr td/test-erasure-code-plugins x 2024-05-08T16:04:00.321 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/test-erasure-code-plugins 2024-05-08T16:04:00.321 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-05-08T16:04:00.322 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-05-08T16:04:00.322 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-05-08T16:04:00.322 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/test-erasure-code-plugins/x 2024-05-08T16:04:00.322 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-05-08T16:04:01.242 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-05-08T16:04:01.242 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-08T16:04:01.242 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-08T16:04:01.243 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-08T16:04:01.243 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T16:04:01.243 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20299 2024-05-08T16:04:01.243 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.20299/$cluster-$name.asok' 2024-05-08T16:04:01.244 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-05-08T16:04:01.245 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-code-plugins/x '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.20299/$cluster-$name.asok' --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-05-08T16:04:01.275 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:47: TEST_preload_warning: get_asok_path mon.a 2024-05-08T16:04:01.275 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-05-08T16:04:01.276 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-05-08T16:04:01.276 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-08T16:04:01.276 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T16:04:01.276 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20299 2024-05-08T16:04:01.276 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20299/ceph-mon.a.asok 2024-05-08T16:04:01.276 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:47: TEST_preload_warning: CEPH_ARGS= 2024-05-08T16:04:01.276 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:47: TEST_preload_warning: ceph --admin-daemon /tmp/ceph-asok.20299/ceph-mon.a.asok log flush 2024-05-08T16:04:01.372 INFO:tasks.workunit.client.0.smithi172.stdout:{} 2024-05-08T16:04:01.385 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:48: TEST_preload_warning: run_osd td/test-erasure-code-plugins 0 --osd_erasure_code_plugins=shec_sse3 2024-05-08T16:04:01.385 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-code-plugins 2024-05-08T16:04:01.385 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-05-08T16:04:01.385 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-05-08T16:04:01.385 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-05-08T16:04:01.386 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-code-plugins/0 2024-05-08T16:04:01.386 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=2a3a20f9-46fd-4090-ba60-6d102331c696 --auth-supported=none --mon-host=127.0.0.1:17110 ' 2024-05-08T16:04:01.386 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-08T16:04:01.386 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-05-08T16:04:01.386 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-08T16:04:01.386 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-code-plugins/0' 2024-05-08T16:04:01.386 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code-plugins/0/journal' 2024-05-08T16:04:01.386 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-05-08T16:04:01.386 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-05-08T16:04:01.386 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-code-plugins' 2024-05-08T16:04:01.388 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-05-08T16:04:01.389 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-08T16:04:01.389 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-08T16:04:01.389 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-08T16:04:01.389 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T16:04:01.389 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20299 2024-05-08T16:04:01.389 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.20299/$cluster-$name.asok' 2024-05-08T16:04:01.389 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.20299/$cluster-$name.asok' 2024-05-08T16:04:01.389 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-05-08T16:04:01.389 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-05-08T16:04:01.389 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-05-08T16:04:01.389 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-code-plugins/$name.log' 2024-05-08T16:04:01.389 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-code-plugins/$name.pid' 2024-05-08T16:04:01.389 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-08T16:04:01.389 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-08T16:04:01.390 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-08T16:04:01.390 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-08T16:04:01.390 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-05-08T16:04:01.390 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_erasure_code_plugins=shec_sse3 2024-05-08T16:04:01.390 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-code-plugins/0 2024-05-08T16:04:01.390 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-05-08T16:04:01.390 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=8c0cd820-232e-45c7-81ad-4f340ef81558 2024-05-08T16:04:01.391 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 8c0cd820-232e-45c7-81ad-4f340ef81558' 2024-05-08T16:04:01.391 INFO:tasks.workunit.client.0.smithi172.stdout:add osd0 8c0cd820-232e-45c7-81ad-4f340ef81558 2024-05-08T16:04:01.391 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-05-08T16:04:01.405 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBxojtmF8KbGBAAUEOU2TkmNQnb/oQRqyqTgg== 2024-05-08T16:04:01.405 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBxojtmF8KbGBAAUEOU2TkmNQnb/oQRqyqTgg=="}' 2024-05-08T16:04:01.405 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 8c0cd820-232e-45c7-81ad-4f340ef81558 -i td/test-erasure-code-plugins/0/new.json 2024-05-08T16:04:01.688 INFO:tasks.workunit.client.0.smithi172.stdout:0 2024-05-08T16:04:01.704 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-code-plugins/0/new.json 2024-05-08T16:04:01.704 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=2a3a20f9-46fd-4090-ba60-6d102331c696 --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.20299/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_erasure_code_plugins=shec_sse3 --mkfs --key AQBxojtmF8KbGBAAUEOU2TkmNQnb/oQRqyqTgg== --osd-uuid 8c0cd820-232e-45c7-81ad-4f340ef81558 2024-05-08T16:04:01.736 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:04:01.741+0000 7fb3504357c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:04:01.748 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:04:01.753+0000 7fb3504357c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:04:01.749 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:04:01.753+0000 7fb3504357c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:04:01.750 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:04:01.753+0000 7fb3504357c0 -1 bluestore(td/test-erasure-code-plugins/0/block) _read_bdev_label failed to open td/test-erasure-code-plugins/0/block: (2) No such file or directory 2024-05-08T16:04:01.750 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:04:01.753+0000 7fb3504357c0 -1 bluestore(td/test-erasure-code-plugins/0/block) _read_bdev_label failed to open td/test-erasure-code-plugins/0/block: (2) No such file or directory 2024-05-08T16:04:01.750 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:04:01.753+0000 7fb3504357c0 -1 bluestore(td/test-erasure-code-plugins/0/block) _read_bdev_label failed to open td/test-erasure-code-plugins/0/block: (2) No such file or directory 2024-05-08T16:04:01.789 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:04:01.793+0000 7fb3504357c0 -1 bluestore(td/test-erasure-code-plugins/0) _read_fsid unparsable uuid 2024-05-08T16:04:08.048 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-code-plugins/0/keyring 2024-05-08T16:04:08.049 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-05-08T16:04:08.050 INFO:tasks.workunit.client.0.smithi172.stdout:adding osd0 key to auth repository 2024-05-08T16:04:08.050 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-05-08T16:04:08.050 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-code-plugins/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-05-08T16:04:08.588 INFO:tasks.workunit.client.0.smithi172.stdout:start osd.0 2024-05-08T16:04:08.589 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-05-08T16:04:08.589 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-05-08T16:04:08.589 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-05-08T16:04:08.589 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-05-08T16:04:08.589 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=2a3a20f9-46fd-4090-ba60-6d102331c696 --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.20299/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_erasure_code_plugins=shec_sse3 2024-05-08T16:04:08.619 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:04:08.625+0000 7fc9f118b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:04:08.621 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:04:08.625+0000 7fc9f118b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:04:08.622 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:04:08.625+0000 7fc9f118b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:04:08.990 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-05-08T16:04:08.991 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-08T16:04:08.991 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-05-08T16:04:08.991 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-08T16:04:08.991 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-08T16:04:08.991 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:04:08.991 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-08T16:04:08.991 INFO:tasks.workunit.client.0.smithi172.stdout:0 2024-05-08T16:04:08.992 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:04:08.992 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:04:09.184 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:04:09.189+0000 7fc9f118b7c0 -1 Falling back to public interface 2024-05-08T16:04:09.395 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:04:10.398 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:04:10.398 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:04:10.398 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-08T16:04:10.399 INFO:tasks.workunit.client.0.smithi172.stdout:1 2024-05-08T16:04:10.399 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:04:10.399 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:04:10.800 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:04:11.802 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:04:11.802 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:04:11.802 INFO:tasks.workunit.client.0.smithi172.stdout:2 2024-05-08T16:04:11.803 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-08T16:04:11.803 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:04:11.803 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:04:11.832 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:04:11.836+0000 7fc9f118b7c0 -1 osd.0 0 log_to_monitors true 2024-05-08T16:04:12.345 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:04:13.347 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:04:13.347 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:04:13.348 INFO:tasks.workunit.client.0.smithi172.stdout:3 2024-05-08T16:04:13.348 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-05-08T16:04:13.348 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:04:13.348 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:04:13.834 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:04:14.835 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:04:14.836 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:04:14.836 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-05-08T16:04:14.836 INFO:tasks.workunit.client.0.smithi172.stdout:4 2024-05-08T16:04:14.836 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:04:14.836 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:04:15.250 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:04:16.252 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:04:16.252 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:04:16.252 INFO:tasks.workunit.client.0.smithi172.stdout:5 2024-05-08T16:04:16.253 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-05-08T16:04:16.253 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:04:16.253 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:04:16.793 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:04:17.795 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:04:17.795 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:04:17.795 INFO:tasks.workunit.client.0.smithi172.stdout:6 2024-05-08T16:04:17.796 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-05-08T16:04:17.796 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:04:17.796 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:04:18.232 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:04:19.233 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:04:19.234 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:04:19.234 INFO:tasks.workunit.client.0.smithi172.stdout:7 2024-05-08T16:04:19.234 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-05-08T16:04:19.234 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:04:19.235 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:04:19.797 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:04:20.798 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:04:20.799 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:04:20.799 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-05-08T16:04:20.799 INFO:tasks.workunit.client.0.smithi172.stdout:8 2024-05-08T16:04:20.799 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:04:20.800 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:04:21.227 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:04:22.229 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:04:22.229 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:04:22.229 INFO:tasks.workunit.client.0.smithi172.stdout:9 2024-05-08T16:04:22.230 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-05-08T16:04:22.230 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:04:22.230 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:04:22.706 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:04:23.708 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:04:23.708 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:04:23.708 INFO:tasks.workunit.client.0.smithi172.stdout:10 2024-05-08T16:04:23.709 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-05-08T16:04:23.709 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:04:23.709 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:04:24.162 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:04:25.164 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:04:25.164 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:04:25.164 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-05-08T16:04:25.164 INFO:tasks.workunit.client.0.smithi172.stdout:11 2024-05-08T16:04:25.164 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:04:25.165 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:04:25.713 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:04:26.714 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:04:26.715 INFO:tasks.workunit.client.0.smithi172.stdout:12 2024-05-08T16:04:26.715 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:04:26.715 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-05-08T16:04:26.716 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:04:26.716 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:04:27.674 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:04:28.676 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:04:28.676 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:04:28.676 INFO:tasks.workunit.client.0.smithi172.stdout:13 2024-05-08T16:04:28.677 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-05-08T16:04:28.677 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:04:28.677 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:04:29.687 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:04:30.689 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:04:30.689 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:04:30.689 INFO:tasks.workunit.client.0.smithi172.stdout:14 2024-05-08T16:04:30.690 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-05-08T16:04:30.690 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:04:30.690 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:04:31.863 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:04:32.865 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:04:32.865 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:04:32.865 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-05-08T16:04:32.865 INFO:tasks.workunit.client.0.smithi172.stdout:15 2024-05-08T16:04:32.865 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:04:32.866 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:04:33.747 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:04:34.749 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:04:34.749 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:04:34.750 INFO:tasks.workunit.client.0.smithi172.stdout:16 2024-05-08T16:04:34.750 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-05-08T16:04:34.750 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:04:34.750 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:04:35.651 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:04:36.652 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:04:36.652 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:04:36.652 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-05-08T16:04:36.652 INFO:tasks.workunit.client.0.smithi172.stdout:17 2024-05-08T16:04:36.653 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:04:36.653 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:04:37.078 INFO:tasks.workunit.client.0.smithi172.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1657952015,v1:127.0.0.1:6803/1657952015] [v2:127.0.0.1:6804/1657952015,v1:127.0.0.1:6805/1657952015] exists,up 8c0cd820-232e-45c7-81ad-4f340ef81558 2024-05-08T16:04:37.079 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-08T16:04:37.080 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-08T16:04:37.080 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-08T16:04:37.080 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:49: TEST_preload_warning: get_asok_path osd.0 2024-05-08T16:04:37.080 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-05-08T16:04:37.080 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-05-08T16:04:37.080 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-08T16:04:37.080 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T16:04:37.080 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20299 2024-05-08T16:04:37.081 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20299/ceph-osd.0.asok 2024-05-08T16:04:37.081 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:49: TEST_preload_warning: CEPH_ARGS= 2024-05-08T16:04:37.081 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:49: TEST_preload_warning: ceph --admin-daemon /tmp/ceph-asok.20299/ceph-osd.0.asok log flush 2024-05-08T16:04:37.176 INFO:tasks.workunit.client.0.smithi172.stdout:{} 2024-05-08T16:04:37.188 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:50: TEST_preload_warning: grep 'WARNING: osd_erasure_code_plugins contains plugin shec_sse3' td/test-erasure-code-plugins/mon.a.log 2024-05-08T16:04:37.192 INFO:tasks.workunit.client.0.smithi172.stdout:2024-05-08T16:03:59.913+0000 7fd68039cc80 0 WARNING: osd_erasure_code_plugins contains plugin shec_sse3 that is now deprecated. Please modify the value for osd_erasure_code_plugins to use shec instead. 2024-05-08T16:04:37.192 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:51: TEST_preload_warning: grep 'WARNING: osd_erasure_code_plugins contains plugin shec_sse3' td/test-erasure-code-plugins/osd.0.log 2024-05-08T16:04:37.195 INFO:tasks.workunit.client.0.smithi172.stdout:2024-05-08T16:04:09.189+0000 7fc9f118b7c0 0 WARNING: osd_erasure_code_plugins contains plugin shec_sse3 that is now deprecated. Please modify the value for osd_erasure_code_plugins to use shec instead. 2024-05-08T16:04:37.195 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:52: TEST_preload_warning: teardown td/test-erasure-code-plugins 2024-05-08T16:04:37.195 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-code-plugins 2024-05-08T16:04:37.195 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-05-08T16:04:37.196 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-code-plugins KILL 2024-05-08T16:04:37.196 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-08T16:04:37.196 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-08T16:04:37.197 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-08T16:04:37.197 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-08T16:04:37.197 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-08T16:04:37.305 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-08T16:04:37.305 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-05-08T16:04:37.306 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-05-08T16:04:37.306 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-05-08T16:04:37.307 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-05-08T16:04:37.307 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-05-08T16:04:37.307 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-05-08T16:04:37.308 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-08T16:04:37.308 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-05-08T16:04:37.308 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-05-08T16:04:37.309 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-08T16:04:37.309 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-05-08T16:04:37.310 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-05-08T16:04:37.310 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-code-plugins 2024-05-08T16:04:37.315 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-05-08T16:04:37.315 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T16:04:37.315 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20299 2024-05-08T16:04:37.316 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.20299 2024-05-08T16:04:37.316 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-05-08T16:04:37.317 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-05-08T16:04:37.317 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:43: TEST_preload_warning: for plugin in ${legacy_jerasure_plugins[*]} ${legacy_shec_plugins[*]} 2024-05-08T16:04:37.317 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:44: TEST_preload_warning: setup td/test-erasure-code-plugins 2024-05-08T16:04:37.317 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/test-erasure-code-plugins 2024-05-08T16:04:37.317 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/test-erasure-code-plugins 2024-05-08T16:04:37.317 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-code-plugins 2024-05-08T16:04:37.317 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-05-08T16:04:37.317 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-code-plugins KILL 2024-05-08T16:04:37.318 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-08T16:04:37.318 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-08T16:04:37.318 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-08T16:04:37.318 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-08T16:04:37.318 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-08T16:04:37.319 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-08T16:04:37.319 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-05-08T16:04:37.320 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-05-08T16:04:37.320 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-05-08T16:04:37.321 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-05-08T16:04:37.321 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-05-08T16:04:37.322 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-05-08T16:04:37.322 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-08T16:04:37.322 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-05-08T16:04:37.323 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-05-08T16:04:37.323 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-08T16:04:37.323 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-05-08T16:04:37.324 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-05-08T16:04:37.324 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-code-plugins 2024-05-08T16:04:37.326 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-05-08T16:04:37.326 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T16:04:37.326 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20299 2024-05-08T16:04:37.326 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.20299 2024-05-08T16:04:37.326 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-05-08T16:04:37.327 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-05-08T16:04:37.327 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/test-erasure-code-plugins 2024-05-08T16:04:37.328 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-05-08T16:04:37.328 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T16:04:37.328 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20299 2024-05-08T16:04:37.328 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.20299 2024-05-08T16:04:37.330 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-05-08T16:04:37.330 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-05-08T16:04:37.330 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-05-08T16:04:37.330 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/test-erasure-code-plugins 1' TERM HUP INT 2024-05-08T16:04:37.330 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:45: TEST_preload_warning: run_mon td/test-erasure-code-plugins a --osd_erasure_code_plugins=shec_sse4 2024-05-08T16:04:37.330 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/test-erasure-code-plugins 2024-05-08T16:04:37.330 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-05-08T16:04:37.330 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-05-08T16:04:37.330 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-05-08T16:04:37.330 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/test-erasure-code-plugins/a 2024-05-08T16:04:37.331 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-code-plugins/a --run-dir=td/test-erasure-code-plugins --osd_erasure_code_plugins=shec_sse4 2024-05-08T16:04:37.738 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-05-08T16:04:37.738 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-08T16:04:37.738 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-08T16:04:37.738 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-08T16:04:37.738 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T16:04:37.738 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20299 2024-05-08T16:04:37.739 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.20299/$cluster-$name.asok' 2024-05-08T16:04:37.739 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-code-plugins/a '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.20299/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-code-plugins/log --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_erasure_code_plugins=shec_sse4 2024-05-08T16:04:37.948 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-05-08T16:04:37.949 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-05-08T16:04:37.950 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-05-08T16:04:37.950 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-05-08T16:04:37.950 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-05-08T16:04:37.951 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-05-08T16:04:37.951 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-05-08T16:04:37.951 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-05-08T16:04:37.951 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-05-08T16:04:37.951 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-08T16:04:37.951 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T16:04:37.951 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20299 2024-05-08T16:04:37.951 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20299/ceph-mon.a.asok 2024-05-08T16:04:37.952 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-05-08T16:04:37.952 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.20299/ceph-mon.a.asok config get fsid 2024-05-08T16:04:38.069 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-05-08T16:04:38.070 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-05-08T16:04:38.070 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-05-08T16:04:38.070 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-05-08T16:04:38.070 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-05-08T16:04:38.071 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-05-08T16:04:38.071 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-05-08T16:04:38.071 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-05-08T16:04:38.071 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-08T16:04:38.071 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T16:04:38.071 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20299 2024-05-08T16:04:38.071 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20299/ceph-mon.a.asok 2024-05-08T16:04:38.072 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-05-08T16:04:38.072 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.20299/ceph-mon.a.asok config get mon_host 2024-05-08T16:04:38.180 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:46: TEST_preload_warning: run_mgr td/test-erasure-code-plugins x 2024-05-08T16:04:38.180 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/test-erasure-code-plugins 2024-05-08T16:04:38.181 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-05-08T16:04:38.181 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-05-08T16:04:38.181 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-05-08T16:04:38.181 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/test-erasure-code-plugins/x 2024-05-08T16:04:38.181 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-05-08T16:04:39.093 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-05-08T16:04:39.094 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-08T16:04:39.094 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-08T16:04:39.094 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-08T16:04:39.094 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T16:04:39.094 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20299 2024-05-08T16:04:39.094 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.20299/$cluster-$name.asok' 2024-05-08T16:04:39.095 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-05-08T16:04:39.095 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-code-plugins/x '--log-file=td/test-erasure-code-plugins/$name.log' '--admin-socket=/tmp/ceph-asok.20299/$cluster-$name.asok' --run-dir=td/test-erasure-code-plugins '--pid-file=td/test-erasure-code-plugins/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-05-08T16:04:39.128 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:47: TEST_preload_warning: get_asok_path mon.a 2024-05-08T16:04:39.128 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-05-08T16:04:39.128 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-05-08T16:04:39.128 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-08T16:04:39.128 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T16:04:39.128 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20299 2024-05-08T16:04:39.128 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20299/ceph-mon.a.asok 2024-05-08T16:04:39.129 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:47: TEST_preload_warning: CEPH_ARGS= 2024-05-08T16:04:39.129 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:47: TEST_preload_warning: ceph --admin-daemon /tmp/ceph-asok.20299/ceph-mon.a.asok log flush 2024-05-08T16:04:39.235 INFO:tasks.workunit.client.0.smithi172.stdout:{} 2024-05-08T16:04:39.251 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:48: TEST_preload_warning: run_osd td/test-erasure-code-plugins 0 --osd_erasure_code_plugins=shec_sse4 2024-05-08T16:04:39.251 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-code-plugins 2024-05-08T16:04:39.251 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-05-08T16:04:39.251 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-05-08T16:04:39.252 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-05-08T16:04:39.252 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-code-plugins/0 2024-05-08T16:04:39.252 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=2a3a20f9-46fd-4090-ba60-6d102331c696 --auth-supported=none --mon-host=127.0.0.1:17110 ' 2024-05-08T16:04:39.252 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-08T16:04:39.252 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-05-08T16:04:39.252 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-08T16:04:39.252 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-code-plugins/0' 2024-05-08T16:04:39.252 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code-plugins/0/journal' 2024-05-08T16:04:39.252 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-05-08T16:04:39.252 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-05-08T16:04:39.252 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-code-plugins' 2024-05-08T16:04:39.252 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-05-08T16:04:39.252 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-08T16:04:39.252 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-08T16:04:39.252 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-08T16:04:39.253 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T16:04:39.253 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20299 2024-05-08T16:04:39.253 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.20299/$cluster-$name.asok' 2024-05-08T16:04:39.255 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.20299/$cluster-$name.asok' 2024-05-08T16:04:39.255 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-05-08T16:04:39.255 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-05-08T16:04:39.255 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-05-08T16:04:39.255 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-code-plugins/$name.log' 2024-05-08T16:04:39.255 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-code-plugins/$name.pid' 2024-05-08T16:04:39.255 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-08T16:04:39.255 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-08T16:04:39.255 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-08T16:04:39.255 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-08T16:04:39.256 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-05-08T16:04:39.256 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_erasure_code_plugins=shec_sse4 2024-05-08T16:04:39.256 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-code-plugins/0 2024-05-08T16:04:39.256 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-05-08T16:04:39.256 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=078caffc-dcda-4c1e-9155-388b51c9f53c 2024-05-08T16:04:39.256 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 078caffc-dcda-4c1e-9155-388b51c9f53c' 2024-05-08T16:04:39.256 INFO:tasks.workunit.client.0.smithi172.stdout:add osd0 078caffc-dcda-4c1e-9155-388b51c9f53c 2024-05-08T16:04:39.256 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-05-08T16:04:39.271 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCXojtm6gKiEBAALoarksirlCRzofUZiwa66Q== 2024-05-08T16:04:39.271 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCXojtm6gKiEBAALoarksirlCRzofUZiwa66Q=="}' 2024-05-08T16:04:39.271 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 078caffc-dcda-4c1e-9155-388b51c9f53c -i td/test-erasure-code-plugins/0/new.json 2024-05-08T16:04:39.584 INFO:tasks.workunit.client.0.smithi172.stdout:0 2024-05-08T16:04:39.600 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-code-plugins/0/new.json 2024-05-08T16:04:39.601 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=2a3a20f9-46fd-4090-ba60-6d102331c696 --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.20299/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_erasure_code_plugins=shec_sse4 --mkfs --key AQCXojtm6gKiEBAALoarksirlCRzofUZiwa66Q== --osd-uuid 078caffc-dcda-4c1e-9155-388b51c9f53c 2024-05-08T16:04:39.638 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:04:39.640+0000 7fed2c6d87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:04:39.662 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:04:39.664+0000 7fed2c6d87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:04:39.663 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:04:39.668+0000 7fed2c6d87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:04:39.663 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:04:39.668+0000 7fed2c6d87c0 -1 bluestore(td/test-erasure-code-plugins/0/block) _read_bdev_label failed to open td/test-erasure-code-plugins/0/block: (2) No such file or directory 2024-05-08T16:04:39.663 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:04:39.668+0000 7fed2c6d87c0 -1 bluestore(td/test-erasure-code-plugins/0/block) _read_bdev_label failed to open td/test-erasure-code-plugins/0/block: (2) No such file or directory 2024-05-08T16:04:39.663 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:04:39.668+0000 7fed2c6d87c0 -1 bluestore(td/test-erasure-code-plugins/0/block) _read_bdev_label failed to open td/test-erasure-code-plugins/0/block: (2) No such file or directory 2024-05-08T16:04:39.701 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:04:39.704+0000 7fed2c6d87c0 -1 bluestore(td/test-erasure-code-plugins/0) _read_fsid unparsable uuid 2024-05-08T16:04:46.015 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-code-plugins/0/keyring 2024-05-08T16:04:46.015 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-05-08T16:04:46.017 INFO:tasks.workunit.client.0.smithi172.stdout:adding osd0 key to auth repository 2024-05-08T16:04:46.017 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-05-08T16:04:46.017 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-code-plugins/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-05-08T16:04:46.578 INFO:tasks.workunit.client.0.smithi172.stdout:start osd.0 2024-05-08T16:04:46.578 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-05-08T16:04:46.578 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=2a3a20f9-46fd-4090-ba60-6d102331c696 --auth-supported=none --mon-host=127.0.0.1:17110 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code-plugins/0 --osd-journal=td/test-erasure-code-plugins/0/journal --chdir= --run-dir=td/test-erasure-code-plugins '--admin-socket=/tmp/ceph-asok.20299/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code-plugins/$name.log' '--pid-file=td/test-erasure-code-plugins/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_erasure_code_plugins=shec_sse4 2024-05-08T16:04:46.579 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-05-08T16:04:46.579 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-05-08T16:04:46.579 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-05-08T16:04:46.607 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:04:46.612+0000 7fd5445337c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:04:46.610 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:04:46.612+0000 7fd5445337c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:04:46.611 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:04:46.616+0000 7fd5445337c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:04:47.056 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-05-08T16:04:47.057 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-08T16:04:47.057 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-05-08T16:04:47.057 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-08T16:04:47.057 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-08T16:04:47.057 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:04:47.057 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-08T16:04:47.057 INFO:tasks.workunit.client.0.smithi172.stdout:0 2024-05-08T16:04:47.057 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:04:47.058 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:04:47.188 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:04:47.192+0000 7fd5445337c0 -1 Falling back to public interface 2024-05-08T16:04:47.456 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:04:48.458 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:04:48.458 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:04:48.458 INFO:tasks.workunit.client.0.smithi172.stdout:1 2024-05-08T16:04:48.459 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-08T16:04:48.459 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:04:48.459 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:04:48.874 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:04:49.875 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:04:49.875 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:04:49.876 INFO:tasks.workunit.client.0.smithi172.stdout:2 2024-05-08T16:04:49.876 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-08T16:04:49.876 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:04:49.876 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:04:49.880 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:04:49.884+0000 7fd5445337c0 -1 osd.0 0 log_to_monitors true 2024-05-08T16:04:50.388 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:04:51.390 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:04:51.390 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:04:51.390 INFO:tasks.workunit.client.0.smithi172.stdout:3 2024-05-08T16:04:51.391 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-05-08T16:04:51.391 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:04:51.391 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:04:51.852 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:04:52.855 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:04:52.855 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:04:52.855 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-05-08T16:04:52.855 INFO:tasks.workunit.client.0.smithi172.stdout:4 2024-05-08T16:04:52.855 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:04:52.856 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:04:53.259 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:04:54.262 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:04:54.262 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:04:54.263 INFO:tasks.workunit.client.0.smithi172.stdout:5 2024-05-08T16:04:54.263 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-05-08T16:04:54.263 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:04:54.263 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:04:54.748 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:04:55.750 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:04:55.750 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:04:55.751 INFO:tasks.workunit.client.0.smithi172.stdout:6 2024-05-08T16:04:55.752 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-05-08T16:04:55.752 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:04:55.752 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:04:56.195 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:04:57.197 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:04:57.197 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:04:57.197 INFO:tasks.workunit.client.0.smithi172.stdout:7 2024-05-08T16:04:57.198 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-05-08T16:04:57.198 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:04:57.198 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:04:57.756 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:04:58.758 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:04:58.758 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:04:58.758 INFO:tasks.workunit.client.0.smithi172.stdout:8 2024-05-08T16:04:58.759 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-05-08T16:04:58.759 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:04:58.759 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:04:59.203 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:05:00.205 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:05:00.205 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:05:00.206 INFO:tasks.workunit.client.0.smithi172.stdout:9 2024-05-08T16:05:00.206 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-05-08T16:05:00.206 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:05:00.206 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:05:00.680 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:05:01.682 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:05:01.682 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:05:01.683 INFO:tasks.workunit.client.0.smithi172.stdout:10 2024-05-08T16:05:01.683 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-05-08T16:05:01.684 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:05:01.684 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:05:02.154 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:05:03.155 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:05:03.156 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:05:03.156 INFO:tasks.workunit.client.0.smithi172.stdout:11 2024-05-08T16:05:03.156 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-05-08T16:05:03.156 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:05:03.157 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:05:03.622 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:05:04.624 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:05:04.624 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:05:04.625 INFO:tasks.workunit.client.0.smithi172.stdout:12 2024-05-08T16:05:04.625 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-05-08T16:05:04.625 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:05:04.625 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:05:05.454 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:05:06.455 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:05:06.456 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:05:06.456 INFO:tasks.workunit.client.0.smithi172.stdout:13 2024-05-08T16:05:06.456 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-05-08T16:05:06.457 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:05:06.457 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:05:07.754 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:05:08.755 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:05:08.755 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:05:08.756 INFO:tasks.workunit.client.0.smithi172.stdout:14 2024-05-08T16:05:08.756 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-05-08T16:05:08.756 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:05:08.756 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:05:09.322 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:05:10.323 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:05:10.323 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:05:10.324 INFO:tasks.workunit.client.0.smithi172.stdout:15 2024-05-08T16:05:10.324 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-05-08T16:05:10.324 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:05:10.324 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:05:10.964 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:05:11.966 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:05:11.966 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:05:11.966 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-05-08T16:05:11.966 INFO:tasks.workunit.client.0.smithi172.stdout:16 2024-05-08T16:05:11.967 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:05:11.967 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:05:12.737 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:05:13.739 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:05:13.739 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:05:13.739 INFO:tasks.workunit.client.0.smithi172.stdout:17 2024-05-08T16:05:13.740 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-05-08T16:05:13.740 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:05:13.740 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:05:14.254 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:05:15.255 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:05:15.255 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:05:15.256 INFO:tasks.workunit.client.0.smithi172.stdout:18 2024-05-08T16:05:15.256 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-05-08T16:05:15.256 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:05:15.256 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:05:15.721 INFO:tasks.workunit.client.0.smithi172.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/778215627,v1:127.0.0.1:6803/778215627] [v2:127.0.0.1:6804/778215627,v1:127.0.0.1:6805/778215627] exists,up 078caffc-dcda-4c1e-9155-388b51c9f53c 2024-05-08T16:05:15.722 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-08T16:05:15.722 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-08T16:05:15.723 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-08T16:05:15.723 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:49: TEST_preload_warning: get_asok_path osd.0 2024-05-08T16:05:15.723 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-05-08T16:05:15.723 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-05-08T16:05:15.723 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-08T16:05:15.723 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T16:05:15.723 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20299 2024-05-08T16:05:15.723 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20299/ceph-osd.0.asok 2024-05-08T16:05:15.724 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:49: TEST_preload_warning: CEPH_ARGS= 2024-05-08T16:05:15.724 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:49: TEST_preload_warning: ceph --admin-daemon /tmp/ceph-asok.20299/ceph-osd.0.asok log flush 2024-05-08T16:05:15.819 INFO:tasks.workunit.client.0.smithi172.stdout:{} 2024-05-08T16:05:15.831 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:50: TEST_preload_warning: grep 'WARNING: osd_erasure_code_plugins contains plugin shec_sse4' td/test-erasure-code-plugins/mon.a.log 2024-05-08T16:05:15.835 INFO:tasks.workunit.client.0.smithi172.stdout:2024-05-08T16:04:37.780+0000 7ffb1207ac80 0 WARNING: osd_erasure_code_plugins contains plugin shec_sse4 that is now deprecated. Please modify the value for osd_erasure_code_plugins to use shec instead. 2024-05-08T16:05:15.836 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:51: TEST_preload_warning: grep 'WARNING: osd_erasure_code_plugins contains plugin shec_sse4' td/test-erasure-code-plugins/osd.0.log 2024-05-08T16:05:15.838 INFO:tasks.workunit.client.0.smithi172.stdout:2024-05-08T16:04:47.192+0000 7fd5445337c0 0 WARNING: osd_erasure_code_plugins contains plugin shec_sse4 that is now deprecated. Please modify the value for osd_erasure_code_plugins to use shec instead. 2024-05-08T16:05:15.840 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:52: TEST_preload_warning: teardown td/test-erasure-code-plugins 2024-05-08T16:05:15.840 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-code-plugins 2024-05-08T16:05:15.840 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-05-08T16:05:15.840 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-code-plugins KILL 2024-05-08T16:05:15.841 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-08T16:05:15.841 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-08T16:05:15.841 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-08T16:05:15.841 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-08T16:05:15.841 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-08T16:05:15.948 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-08T16:05:15.948 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-05-08T16:05:15.949 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-05-08T16:05:15.949 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-05-08T16:05:15.950 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-05-08T16:05:15.950 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-05-08T16:05:15.950 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-05-08T16:05:15.951 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-08T16:05:15.951 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-05-08T16:05:15.954 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-05-08T16:05:15.954 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-08T16:05:15.954 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-05-08T16:05:15.954 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-05-08T16:05:15.954 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-code-plugins 2024-05-08T16:05:15.960 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-05-08T16:05:15.961 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T16:05:15.961 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20299 2024-05-08T16:05:15.961 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.20299 2024-05-08T16:05:15.962 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-05-08T16:05:15.962 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-05-08T16:05:15.963 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code-plugins.sh:54: TEST_preload_warning: return 0 2024-05-08T16:05:15.963 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2314: main: code=0 2024-05-08T16:05:15.963 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2318: main: teardown td/test-erasure-code-plugins 0 2024-05-08T16:05:15.963 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-code-plugins 2024-05-08T16:05:15.963 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2024-05-08T16:05:15.963 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-code-plugins KILL 2024-05-08T16:05:15.963 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-08T16:05:15.963 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-08T16:05:15.963 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-08T16:05:15.963 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-08T16:05:15.963 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-08T16:05:15.964 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-08T16:05:15.964 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-05-08T16:05:15.965 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-05-08T16:05:15.965 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-05-08T16:05:15.966 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-05-08T16:05:15.966 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-05-08T16:05:15.967 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-05-08T16:05:15.967 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-08T16:05:15.967 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-05-08T16:05:15.968 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-05-08T16:05:15.968 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-08T16:05:15.969 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-05-08T16:05:15.969 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2024-05-08T16:05:15.970 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-code-plugins 2024-05-08T16:05:15.972 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-05-08T16:05:15.972 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T16:05:15.972 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20299 2024-05-08T16:05:15.972 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.20299 2024-05-08T16:05:15.972 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-05-08T16:05:15.973 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-05-08T16:05:15.973 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2319: main: return 0 2024-05-08T16:05:15.973 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-05-08T16:05:15.973 DEBUG:teuthology.orchestra.run.smithi172:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-05-08T16:05:15.982 INFO:tasks.workunit:Running workunit erasure-code/test-erasure-code.sh... 2024-05-08T16:05:15.982 DEBUG:teuthology.orchestra.run.smithi172:workunit test erasure-code/test-erasure-code.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=95a4ad3c90f2f12f920e87725ad8d9002f948722 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh 2024-05-08T16:05:16.034 INFO:tasks.workunit.client.0.smithi172.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-05-08T16:05:16.039 INFO:tasks.workunit.client.0.smithi172.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-05-08T16:05:16.039 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2307: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-05-08T16:05:16.040 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2307: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-05-08T16:05:16.040 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2308: main: export PYTHONWARNINGS=ignore 2024-05-08T16:05:16.040 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2308: main: PYTHONWARNINGS=ignore 2024-05-08T16:05:16.040 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2309: main: export CEPH_CONF=/dev/null 2024-05-08T16:05:16.040 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2309: main: CEPH_CONF=/dev/null 2024-05-08T16:05:16.040 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2310: main: unset CEPH_ARGS 2024-05-08T16:05:16.040 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2312: main: local code 2024-05-08T16:05:16.040 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2313: main: run td/test-erasure-code 2024-05-08T16:05:16.040 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:22: run: local dir=td/test-erasure-code 2024-05-08T16:05:16.040 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:23: run: shift 2024-05-08T16:05:16.040 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:25: run: export CEPH_MON=127.0.0.1:7101 2024-05-08T16:05:16.040 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:25: run: CEPH_MON=127.0.0.1:7101 2024-05-08T16:05:16.040 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:26: run: export CEPH_ARGS 2024-05-08T16:05:16.040 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:27: run: uuidgen 2024-05-08T16:05:16.042 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:27: run: CEPH_ARGS+='--fsid=f37874d1-6990-4767-8498-03a75cc31118 --auth-supported=none ' 2024-05-08T16:05:16.042 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:28: run: CEPH_ARGS+='--mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false' 2024-05-08T16:05:16.042 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:30: run: setup td/test-erasure-code 2024-05-08T16:05:16.042 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/test-erasure-code 2024-05-08T16:05:16.043 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/test-erasure-code 2024-05-08T16:05:16.043 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-code 2024-05-08T16:05:16.043 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-05-08T16:05:16.043 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-code KILL 2024-05-08T16:05:16.043 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-08T16:05:16.043 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-08T16:05:16.043 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-08T16:05:16.043 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-08T16:05:16.044 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-08T16:05:16.044 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-08T16:05:16.045 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-05-08T16:05:16.045 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-05-08T16:05:16.045 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-05-08T16:05:16.045 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-05-08T16:05:16.045 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-05-08T16:05:16.045 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-05-08T16:05:16.045 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-08T16:05:16.045 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-05-08T16:05:16.046 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-05-08T16:05:16.046 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-08T16:05:16.046 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-05-08T16:05:16.047 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-05-08T16:05:16.047 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-code 2024-05-08T16:05:16.048 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-05-08T16:05:16.049 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T16:05:16.049 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.48735 2024-05-08T16:05:16.049 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.48735 2024-05-08T16:05:16.049 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-05-08T16:05:16.049 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-05-08T16:05:16.049 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/test-erasure-code 2024-05-08T16:05:16.051 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-05-08T16:05:16.051 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T16:05:16.051 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.48735 2024-05-08T16:05:16.051 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.48735 2024-05-08T16:05:16.052 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-05-08T16:05:16.053 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2024-05-08T16:05:16.053 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2024-05-08T16:05:16.054 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-05-08T16:05:16.054 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/test-erasure-code 1' TERM HUP INT 2024-05-08T16:05:16.054 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:31: run: run_mon td/test-erasure-code a 2024-05-08T16:05:16.054 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/test-erasure-code 2024-05-08T16:05:16.054 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-05-08T16:05:16.054 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-05-08T16:05:16.054 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-05-08T16:05:16.054 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/test-erasure-code/a 2024-05-08T16:05:16.054 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-code/a --run-dir=td/test-erasure-code 2024-05-08T16:05:16.453 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-05-08T16:05:16.453 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-08T16:05:16.453 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-08T16:05:16.454 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-08T16:05:16.454 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T16:05:16.454 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.48735 2024-05-08T16:05:16.454 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.48735/$cluster-$name.asok' 2024-05-08T16:05:16.454 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-code/a '--log-file=td/test-erasure-code/$name.log' '--admin-socket=/tmp/ceph-asok.48735/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-code/log --run-dir=td/test-erasure-code '--pid-file=td/test-erasure-code/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-05-08T16:05:16.662 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-05-08T16:05:16.663 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-05-08T16:05:16.663 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-05-08T16:05:16.663 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-05-08T16:05:16.663 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-05-08T16:05:16.664 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-05-08T16:05:16.664 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-05-08T16:05:16.664 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-05-08T16:05:16.664 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-05-08T16:05:16.664 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-08T16:05:16.664 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T16:05:16.665 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.48735 2024-05-08T16:05:16.665 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.48735/ceph-mon.a.asok 2024-05-08T16:05:16.665 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-05-08T16:05:16.665 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.48735/ceph-mon.a.asok config get fsid 2024-05-08T16:05:16.779 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-05-08T16:05:16.779 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-05-08T16:05:16.779 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-05-08T16:05:16.779 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-05-08T16:05:16.779 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-05-08T16:05:16.779 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-05-08T16:05:16.779 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-05-08T16:05:16.779 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-05-08T16:05:16.780 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-08T16:05:16.780 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T16:05:16.780 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.48735 2024-05-08T16:05:16.780 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.48735/ceph-mon.a.asok 2024-05-08T16:05:16.781 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-05-08T16:05:16.781 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.48735/ceph-mon.a.asok config get mon_host 2024-05-08T16:05:16.889 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:32: run: run_mgr td/test-erasure-code x 2024-05-08T16:05:16.889 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/test-erasure-code 2024-05-08T16:05:16.889 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-05-08T16:05:16.889 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-05-08T16:05:16.889 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-05-08T16:05:16.889 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/test-erasure-code/x 2024-05-08T16:05:16.890 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-05-08T16:05:17.809 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-05-08T16:05:17.810 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-08T16:05:17.810 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-08T16:05:17.810 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-08T16:05:17.810 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T16:05:17.810 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.48735 2024-05-08T16:05:17.810 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.48735/$cluster-$name.asok' 2024-05-08T16:05:17.810 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-05-08T16:05:17.811 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-code/x '--log-file=td/test-erasure-code/$name.log' '--admin-socket=/tmp/ceph-asok.48735/$cluster-$name.asok' --run-dir=td/test-erasure-code '--pid-file=td/test-erasure-code/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-05-08T16:05:17.842 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:34: run: get_asok_path mon.a 2024-05-08T16:05:17.842 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-05-08T16:05:17.842 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-05-08T16:05:17.842 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-08T16:05:17.842 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T16:05:17.842 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.48735 2024-05-08T16:05:17.843 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.48735/ceph-mon.a.asok 2024-05-08T16:05:17.843 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:34: run: CEPH_ARGS= 2024-05-08T16:05:17.843 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:34: run: ceph --admin-daemon /tmp/ceph-asok.48735/ceph-mon.a.asok log flush 2024-05-08T16:05:17.937 INFO:tasks.workunit.client.0.smithi172.stdout:{} 2024-05-08T16:05:17.950 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:35: run: grep 'load: jerasure.*lrc' td/test-erasure-code/mon.a.log 2024-05-08T16:05:17.953 INFO:tasks.workunit.client.0.smithi172.stdout:2024-05-08T16:05:16.499+0000 7f4e34ae7c80 0 load: jerasure load: lrc 2024-05-08T16:05:17.954 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:36: run: seq 0 10 2024-05-08T16:05:17.958 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:36: run: for id in $(seq 0 10) 2024-05-08T16:05:17.958 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:37: run: run_osd td/test-erasure-code 0 2024-05-08T16:05:17.958 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-code 2024-05-08T16:05:17.958 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-05-08T16:05:17.958 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-05-08T16:05:17.959 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-05-08T16:05:17.959 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-code/0 2024-05-08T16:05:17.959 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=f37874d1-6990-4767-8498-03a75cc31118 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false' 2024-05-08T16:05:17.959 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-08T16:05:17.959 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-05-08T16:05:17.959 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-08T16:05:17.959 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-code/0' 2024-05-08T16:05:17.959 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code/0/journal' 2024-05-08T16:05:17.959 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-05-08T16:05:17.959 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-05-08T16:05:17.959 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-code' 2024-05-08T16:05:17.960 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-05-08T16:05:17.960 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-08T16:05:17.960 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-08T16:05:17.960 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-08T16:05:17.960 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T16:05:17.960 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.48735 2024-05-08T16:05:17.960 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.48735/$cluster-$name.asok' 2024-05-08T16:05:17.960 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.48735/$cluster-$name.asok' 2024-05-08T16:05:17.960 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-05-08T16:05:17.960 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-05-08T16:05:17.960 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-05-08T16:05:17.961 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-code/$name.log' 2024-05-08T16:05:17.961 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-code/$name.pid' 2024-05-08T16:05:17.961 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-08T16:05:17.961 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-08T16:05:17.961 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-08T16:05:17.961 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-08T16:05:17.961 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-05-08T16:05:17.961 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-05-08T16:05:17.961 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-code/0 2024-05-08T16:05:17.962 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-05-08T16:05:17.962 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=ee5841f7-e628-4071-be29-e1a14e6579c2 2024-05-08T16:05:17.962 INFO:tasks.workunit.client.0.smithi172.stdout:add osd0 ee5841f7-e628-4071-be29-e1a14e6579c2 2024-05-08T16:05:17.963 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 ee5841f7-e628-4071-be29-e1a14e6579c2' 2024-05-08T16:05:17.963 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-05-08T16:05:17.982 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQC9ojtmxkb8OhAA3BMvbzlZOPDsgRgmq0KI6g== 2024-05-08T16:05:17.982 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQC9ojtmxkb8OhAA3BMvbzlZOPDsgRgmq0KI6g=="}' 2024-05-08T16:05:17.982 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new ee5841f7-e628-4071-be29-e1a14e6579c2 -i td/test-erasure-code/0/new.json 2024-05-08T16:05:18.292 INFO:tasks.workunit.client.0.smithi172.stdout:0 2024-05-08T16:05:18.307 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-code/0/new.json 2024-05-08T16:05:18.308 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=f37874d1-6990-4767-8498-03a75cc31118 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/0 --osd-journal=td/test-erasure-code/0/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.48735/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQC9ojtmxkb8OhAA3BMvbzlZOPDsgRgmq0KI6g== --osd-uuid ee5841f7-e628-4071-be29-e1a14e6579c2 2024-05-08T16:05:18.345 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:05:18.347+0000 7f87009e27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:05:18.369 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:05:18.371+0000 7f87009e27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:05:18.370 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:05:18.375+0000 7f87009e27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:05:18.371 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:05:18.375+0000 7f87009e27c0 -1 bluestore(td/test-erasure-code/0/block) _read_bdev_label failed to open td/test-erasure-code/0/block: (2) No such file or directory 2024-05-08T16:05:18.371 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:05:18.375+0000 7f87009e27c0 -1 bluestore(td/test-erasure-code/0/block) _read_bdev_label failed to open td/test-erasure-code/0/block: (2) No such file or directory 2024-05-08T16:05:18.371 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:05:18.375+0000 7f87009e27c0 -1 bluestore(td/test-erasure-code/0/block) _read_bdev_label failed to open td/test-erasure-code/0/block: (2) No such file or directory 2024-05-08T16:05:18.409 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:05:18.411+0000 7f87009e27c0 -1 bluestore(td/test-erasure-code/0) _read_fsid unparsable uuid 2024-05-08T16:05:24.597 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-code/0/keyring 2024-05-08T16:05:24.597 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-05-08T16:05:24.599 INFO:tasks.workunit.client.0.smithi172.stdout:adding osd0 key to auth repository 2024-05-08T16:05:24.599 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-05-08T16:05:24.599 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-code/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-05-08T16:05:25.143 INFO:tasks.workunit.client.0.smithi172.stdout:start osd.0 2024-05-08T16:05:25.144 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-05-08T16:05:25.144 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=f37874d1-6990-4767-8498-03a75cc31118 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/0 --osd-journal=td/test-erasure-code/0/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.48735/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-05-08T16:05:25.144 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-05-08T16:05:25.144 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-05-08T16:05:25.144 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-05-08T16:05:25.173 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:05:25.175+0000 7f44dd1e37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:05:25.199 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:05:25.203+0000 7f44dd1e37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:05:25.200 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:05:25.203+0000 7f44dd1e37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:05:25.547 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-05-08T16:05:25.547 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-08T16:05:25.547 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-05-08T16:05:25.547 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-08T16:05:25.547 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-08T16:05:25.547 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:05:25.547 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-08T16:05:25.547 INFO:tasks.workunit.client.0.smithi172.stdout:0 2024-05-08T16:05:25.548 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:05:25.548 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:05:25.763 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:05:25.767+0000 7f44dd1e37c0 -1 Falling back to public interface 2024-05-08T16:05:25.989 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:05:26.990 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:05:26.990 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:05:26.991 INFO:tasks.workunit.client.0.smithi172.stdout:1 2024-05-08T16:05:26.991 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-08T16:05:26.991 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:05:26.991 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:05:27.388 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:05:28.339 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:05:28.343+0000 7f44dd1e37c0 -1 osd.0 0 log_to_monitors true 2024-05-08T16:05:28.390 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:05:28.390 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:05:28.390 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-08T16:05:28.390 INFO:tasks.workunit.client.0.smithi172.stdout:2 2024-05-08T16:05:28.391 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:05:28.391 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:05:28.978 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:05:29.980 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:05:29.980 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:05:29.980 INFO:tasks.workunit.client.0.smithi172.stdout:3 2024-05-08T16:05:29.981 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-05-08T16:05:29.981 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:05:29.981 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:05:30.439 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:05:31.440 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:05:31.440 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:05:31.441 INFO:tasks.workunit.client.0.smithi172.stdout:4 2024-05-08T16:05:31.441 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-05-08T16:05:31.441 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:05:31.441 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:05:31.872 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:05:32.873 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:05:32.873 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:05:32.873 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-05-08T16:05:32.873 INFO:tasks.workunit.client.0.smithi172.stdout:5 2024-05-08T16:05:32.874 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:05:32.874 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:05:33.499 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:05:34.501 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:05:34.501 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:05:34.501 INFO:tasks.workunit.client.0.smithi172.stdout:6 2024-05-08T16:05:34.502 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-05-08T16:05:34.502 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:05:34.502 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:05:34.930 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:05:35.931 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:05:35.931 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:05:35.931 INFO:tasks.workunit.client.0.smithi172.stdout:7 2024-05-08T16:05:35.932 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-05-08T16:05:35.932 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:05:35.932 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:05:36.339 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:05:37.341 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:05:37.341 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:05:37.341 INFO:tasks.workunit.client.0.smithi172.stdout:8 2024-05-08T16:05:37.342 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-05-08T16:05:37.342 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:05:37.342 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:05:37.780 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:05:38.781 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:05:38.781 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:05:38.781 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-05-08T16:05:38.781 INFO:tasks.workunit.client.0.smithi172.stdout:9 2024-05-08T16:05:38.782 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:05:38.782 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:05:39.278 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:05:40.278 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:05:40.278 INFO:tasks.workunit.client.0.smithi172.stdout:10 2024-05-08T16:05:40.279 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:05:40.279 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-05-08T16:05:40.279 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:05:40.280 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:05:40.723 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:05:41.725 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:05:41.725 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:05:41.726 INFO:tasks.workunit.client.0.smithi172.stdout:11 2024-05-08T16:05:41.726 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-05-08T16:05:41.726 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:05:41.726 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:05:42.559 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:05:43.560 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:05:43.561 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:05:43.561 INFO:tasks.workunit.client.0.smithi172.stdout:12 2024-05-08T16:05:43.561 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-05-08T16:05:43.562 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:05:43.562 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:05:44.389 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:05:45.391 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:05:45.391 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:05:45.391 INFO:tasks.workunit.client.0.smithi172.stdout:13 2024-05-08T16:05:45.392 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-05-08T16:05:45.392 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:05:45.392 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:05:46.246 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:05:47.250 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:05:47.251 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:05:47.251 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-05-08T16:05:47.251 INFO:tasks.workunit.client.0.smithi172.stdout:14 2024-05-08T16:05:47.251 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:05:47.251 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:05:48.149 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:05:49.151 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:05:49.151 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:05:49.151 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-05-08T16:05:49.151 INFO:tasks.workunit.client.0.smithi172.stdout:15 2024-05-08T16:05:49.152 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:05:49.152 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:05:49.821 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:05:50.822 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:05:50.823 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:05:50.823 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-05-08T16:05:50.823 INFO:tasks.workunit.client.0.smithi172.stdout:16 2024-05-08T16:05:50.823 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:05:50.823 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:05:51.916 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:05:52.918 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:05:52.918 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:05:52.919 INFO:tasks.workunit.client.0.smithi172.stdout:17 2024-05-08T16:05:52.919 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-05-08T16:05:52.919 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:05:52.920 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:05:53.355 INFO:tasks.workunit.client.0.smithi172.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3299969833,v1:127.0.0.1:6803/3299969833] [v2:127.0.0.1:6804/3299969833,v1:127.0.0.1:6805/3299969833] exists,up ee5841f7-e628-4071-be29-e1a14e6579c2 2024-05-08T16:05:53.359 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-08T16:05:53.359 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-08T16:05:53.359 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-08T16:05:53.359 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:36: run: for id in $(seq 0 10) 2024-05-08T16:05:53.359 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:37: run: run_osd td/test-erasure-code 1 2024-05-08T16:05:53.359 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-code 2024-05-08T16:05:53.359 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-05-08T16:05:53.359 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-05-08T16:05:53.359 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-05-08T16:05:53.359 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-code/1 2024-05-08T16:05:53.360 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=f37874d1-6990-4767-8498-03a75cc31118 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false' 2024-05-08T16:05:53.360 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-08T16:05:53.360 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-05-08T16:05:53.360 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-08T16:05:53.360 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-code/1' 2024-05-08T16:05:53.360 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code/1/journal' 2024-05-08T16:05:53.360 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-05-08T16:05:53.360 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-05-08T16:05:53.360 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-code' 2024-05-08T16:05:53.360 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-05-08T16:05:53.360 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-08T16:05:53.360 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-08T16:05:53.360 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-08T16:05:53.360 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T16:05:53.361 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.48735 2024-05-08T16:05:53.361 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.48735/$cluster-$name.asok' 2024-05-08T16:05:53.361 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.48735/$cluster-$name.asok' 2024-05-08T16:05:53.361 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-05-08T16:05:53.361 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-05-08T16:05:53.361 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-05-08T16:05:53.361 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-code/$name.log' 2024-05-08T16:05:53.361 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-code/$name.pid' 2024-05-08T16:05:53.361 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-08T16:05:53.361 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-08T16:05:53.361 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-08T16:05:53.361 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-08T16:05:53.361 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-05-08T16:05:53.361 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-05-08T16:05:53.361 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-code/1 2024-05-08T16:05:53.362 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-05-08T16:05:53.362 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=d9d86e21-6ab5-469c-a677-01430db48d53 2024-05-08T16:05:53.362 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 d9d86e21-6ab5-469c-a677-01430db48d53' 2024-05-08T16:05:53.362 INFO:tasks.workunit.client.0.smithi172.stdout:add osd1 d9d86e21-6ab5-469c-a677-01430db48d53 2024-05-08T16:05:53.363 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-05-08T16:05:53.377 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDhojtm5k78FhAApY4AN2xAwe/9NNGut85chw== 2024-05-08T16:05:53.377 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDhojtm5k78FhAApY4AN2xAwe/9NNGut85chw=="}' 2024-05-08T16:05:53.377 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new d9d86e21-6ab5-469c-a677-01430db48d53 -i td/test-erasure-code/1/new.json 2024-05-08T16:05:53.943 INFO:tasks.workunit.client.0.smithi172.stdout:1 2024-05-08T16:05:53.959 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-code/1/new.json 2024-05-08T16:05:53.960 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=f37874d1-6990-4767-8498-03a75cc31118 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/1 --osd-journal=td/test-erasure-code/1/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.48735/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDhojtm5k78FhAApY4AN2xAwe/9NNGut85chw== --osd-uuid d9d86e21-6ab5-469c-a677-01430db48d53 2024-05-08T16:05:53.994 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:05:54.003+0000 7fb6f5a327c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:05:54.024 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:05:54.031+0000 7fb6f5a327c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:05:54.026 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:05:54.031+0000 7fb6f5a327c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:05:54.026 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:05:54.035+0000 7fb6f5a327c0 -1 bluestore(td/test-erasure-code/1/block) _read_bdev_label failed to open td/test-erasure-code/1/block: (2) No such file or directory 2024-05-08T16:05:54.026 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:05:54.035+0000 7fb6f5a327c0 -1 bluestore(td/test-erasure-code/1/block) _read_bdev_label failed to open td/test-erasure-code/1/block: (2) No such file or directory 2024-05-08T16:05:54.026 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:05:54.035+0000 7fb6f5a327c0 -1 bluestore(td/test-erasure-code/1/block) _read_bdev_label failed to open td/test-erasure-code/1/block: (2) No such file or directory 2024-05-08T16:05:54.064 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:05:54.071+0000 7fb6f5a327c0 -1 bluestore(td/test-erasure-code/1) _read_fsid unparsable uuid 2024-05-08T16:06:00.775 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-code/1/keyring 2024-05-08T16:06:00.775 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-05-08T16:06:00.777 INFO:tasks.workunit.client.0.smithi172.stdout:adding osd1 key to auth repository 2024-05-08T16:06:00.777 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-05-08T16:06:00.777 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-code/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-05-08T16:06:01.335 INFO:tasks.workunit.client.0.smithi172.stdout:start osd.1 2024-05-08T16:06:01.336 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-05-08T16:06:01.336 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=f37874d1-6990-4767-8498-03a75cc31118 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/1 --osd-journal=td/test-erasure-code/1/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.48735/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-05-08T16:06:01.336 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-05-08T16:06:01.336 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-05-08T16:06:01.336 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-05-08T16:06:01.365 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:06:01.371+0000 7f89f3d667c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:06:01.367 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:06:01.375+0000 7f89f3d667c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:06:01.368 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:06:01.375+0000 7f89f3d667c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:06:01.767 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-05-08T16:06:01.767 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-08T16:06:01.767 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-05-08T16:06:01.767 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-08T16:06:01.767 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-08T16:06:01.767 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:06:01.768 INFO:tasks.workunit.client.0.smithi172.stdout:0 2024-05-08T16:06:01.769 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-08T16:06:01.769 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:06:01.769 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T16:06:01.927 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:06:01.935+0000 7f89f3d667c0 -1 Falling back to public interface 2024-05-08T16:06:02.197 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:06:03.198 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:06:03.198 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:06:03.199 INFO:tasks.workunit.client.0.smithi172.stdout:1 2024-05-08T16:06:03.200 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-08T16:06:03.200 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:06:03.200 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T16:06:03.614 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:06:04.586 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:06:04.595+0000 7f89f3d667c0 -1 osd.1 0 log_to_monitors true 2024-05-08T16:06:04.616 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:06:04.616 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:06:04.617 INFO:tasks.workunit.client.0.smithi172.stdout:2 2024-05-08T16:06:04.617 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-08T16:06:04.617 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:06:04.617 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T16:06:05.346 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:06:06.347 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:06:06.347 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:06:06.347 INFO:tasks.workunit.client.0.smithi172.stdout:3 2024-05-08T16:06:06.348 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-05-08T16:06:06.348 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:06:06.348 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T16:06:06.840 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:06:07.841 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:06:07.841 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:06:07.841 INFO:tasks.workunit.client.0.smithi172.stdout:4 2024-05-08T16:06:07.842 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-05-08T16:06:07.842 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:06:07.842 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T16:06:08.289 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:06:09.290 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:06:09.290 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:06:09.291 INFO:tasks.workunit.client.0.smithi172.stdout:5 2024-05-08T16:06:09.291 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-05-08T16:06:09.291 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:06:09.291 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T16:06:09.812 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:06:10.814 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:06:10.814 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:06:10.814 INFO:tasks.workunit.client.0.smithi172.stdout:6 2024-05-08T16:06:10.815 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-05-08T16:06:10.815 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:06:10.815 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T16:06:11.284 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:06:12.285 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:06:12.286 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:06:12.286 INFO:tasks.workunit.client.0.smithi172.stdout:7 2024-05-08T16:06:12.286 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-05-08T16:06:12.286 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:06:12.287 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T16:06:12.838 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:06:13.839 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:06:13.840 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:06:13.840 INFO:tasks.workunit.client.0.smithi172.stdout:8 2024-05-08T16:06:13.840 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-05-08T16:06:13.840 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:06:13.840 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T16:06:14.272 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:06:15.273 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:06:15.273 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:06:15.274 INFO:tasks.workunit.client.0.smithi172.stdout:9 2024-05-08T16:06:15.274 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-05-08T16:06:15.274 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:06:15.274 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T16:06:15.746 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:06:16.748 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:06:16.748 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:06:16.749 INFO:tasks.workunit.client.0.smithi172.stdout:10 2024-05-08T16:06:16.749 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-05-08T16:06:16.749 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:06:16.749 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T16:06:17.292 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:06:18.294 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:06:18.294 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:06:18.294 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-05-08T16:06:18.294 INFO:tasks.workunit.client.0.smithi172.stdout:11 2024-05-08T16:06:18.295 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:06:18.295 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T16:06:19.054 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:06:20.055 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:06:20.056 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:06:20.056 INFO:tasks.workunit.client.0.smithi172.stdout:12 2024-05-08T16:06:20.056 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-05-08T16:06:20.057 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:06:20.057 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T16:06:20.598 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:06:21.600 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:06:21.600 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:06:21.601 INFO:tasks.workunit.client.0.smithi172.stdout:13 2024-05-08T16:06:21.601 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-05-08T16:06:21.601 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:06:21.601 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T16:06:22.420 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:06:23.421 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:06:23.422 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:06:23.422 INFO:tasks.workunit.client.0.smithi172.stdout:14 2024-05-08T16:06:23.422 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-05-08T16:06:23.423 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:06:23.423 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T16:06:24.233 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:06:25.235 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:06:25.235 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:06:25.235 INFO:tasks.workunit.client.0.smithi172.stdout:15 2024-05-08T16:06:25.236 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-05-08T16:06:25.236 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:06:25.236 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T16:06:26.068 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:06:27.070 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:06:27.070 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:06:27.071 INFO:tasks.workunit.client.0.smithi172.stdout:16 2024-05-08T16:06:27.071 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-05-08T16:06:27.071 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:06:27.071 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T16:06:27.699 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:06:28.700 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:06:28.700 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:06:28.701 INFO:tasks.workunit.client.0.smithi172.stdout:17 2024-05-08T16:06:28.701 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-05-08T16:06:28.701 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:06:28.701 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T16:06:29.761 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:06:30.763 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:06:30.763 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:06:30.763 INFO:tasks.workunit.client.0.smithi172.stdout:18 2024-05-08T16:06:30.764 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-05-08T16:06:30.764 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:06:30.764 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T16:06:31.202 INFO:tasks.workunit.client.0.smithi172.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/2303430852,v1:127.0.0.1:6811/2303430852] [v2:127.0.0.1:6812/2303430852,v1:127.0.0.1:6813/2303430852] exists,up d9d86e21-6ab5-469c-a677-01430db48d53 2024-05-08T16:06:31.205 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-08T16:06:31.205 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-08T16:06:31.205 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-08T16:06:31.205 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:36: run: for id in $(seq 0 10) 2024-05-08T16:06:31.206 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:37: run: run_osd td/test-erasure-code 2 2024-05-08T16:06:31.206 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-code 2024-05-08T16:06:31.206 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-05-08T16:06:31.206 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-05-08T16:06:31.206 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-05-08T16:06:31.206 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-code/2 2024-05-08T16:06:31.206 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=f37874d1-6990-4767-8498-03a75cc31118 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false' 2024-05-08T16:06:31.206 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-08T16:06:31.206 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-05-08T16:06:31.206 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-08T16:06:31.206 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-code/2' 2024-05-08T16:06:31.206 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code/2/journal' 2024-05-08T16:06:31.207 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-05-08T16:06:31.207 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-05-08T16:06:31.207 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-code' 2024-05-08T16:06:31.207 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-05-08T16:06:31.207 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-08T16:06:31.207 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-08T16:06:31.207 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-08T16:06:31.207 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T16:06:31.207 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.48735 2024-05-08T16:06:31.207 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.48735/$cluster-$name.asok' 2024-05-08T16:06:31.207 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.48735/$cluster-$name.asok' 2024-05-08T16:06:31.207 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-05-08T16:06:31.207 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-05-08T16:06:31.207 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-05-08T16:06:31.207 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-code/$name.log' 2024-05-08T16:06:31.208 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-code/$name.pid' 2024-05-08T16:06:31.208 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-08T16:06:31.208 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-08T16:06:31.208 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-08T16:06:31.208 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-08T16:06:31.208 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-05-08T16:06:31.208 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-05-08T16:06:31.208 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-code/2 2024-05-08T16:06:31.208 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-05-08T16:06:31.209 INFO:tasks.workunit.client.0.smithi172.stdout:add osd2 09f538fa-242a-496b-8a2c-83b2f8a35870 2024-05-08T16:06:31.209 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=09f538fa-242a-496b-8a2c-83b2f8a35870 2024-05-08T16:06:31.209 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 09f538fa-242a-496b-8a2c-83b2f8a35870' 2024-05-08T16:06:31.209 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-05-08T16:06:31.228 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAHoztmdQMoDhAAeG9tPkfEyzVJEMKEsUv2kw== 2024-05-08T16:06:31.228 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAHoztmdQMoDhAAeG9tPkfEyzVJEMKEsUv2kw=="}' 2024-05-08T16:06:31.229 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 09f538fa-242a-496b-8a2c-83b2f8a35870 -i td/test-erasure-code/2/new.json 2024-05-08T16:06:31.812 INFO:tasks.workunit.client.0.smithi172.stdout:2 2024-05-08T16:06:31.827 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-code/2/new.json 2024-05-08T16:06:31.829 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=f37874d1-6990-4767-8498-03a75cc31118 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/2 --osd-journal=td/test-erasure-code/2/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.48735/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAHoztmdQMoDhAAeG9tPkfEyzVJEMKEsUv2kw== --osd-uuid 09f538fa-242a-496b-8a2c-83b2f8a35870 2024-05-08T16:06:31.863 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:06:31.870+0000 7fcc4ad907c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:06:31.881 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:06:31.890+0000 7fcc4ad907c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:06:31.882 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:06:31.890+0000 7fcc4ad907c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:06:31.883 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:06:31.890+0000 7fcc4ad907c0 -1 bluestore(td/test-erasure-code/2/block) _read_bdev_label failed to open td/test-erasure-code/2/block: (2) No such file or directory 2024-05-08T16:06:31.883 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:06:31.890+0000 7fcc4ad907c0 -1 bluestore(td/test-erasure-code/2/block) _read_bdev_label failed to open td/test-erasure-code/2/block: (2) No such file or directory 2024-05-08T16:06:31.883 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:06:31.890+0000 7fcc4ad907c0 -1 bluestore(td/test-erasure-code/2/block) _read_bdev_label failed to open td/test-erasure-code/2/block: (2) No such file or directory 2024-05-08T16:06:31.921 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:06:31.926+0000 7fcc4ad907c0 -1 bluestore(td/test-erasure-code/2) _read_fsid unparsable uuid 2024-05-08T16:06:38.587 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-code/2/keyring 2024-05-08T16:06:38.588 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-05-08T16:06:38.589 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-05-08T16:06:38.589 INFO:tasks.workunit.client.0.smithi172.stdout:adding osd2 key to auth repository 2024-05-08T16:06:38.590 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-code/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-05-08T16:06:39.147 INFO:tasks.workunit.client.0.smithi172.stdout:start osd.2 2024-05-08T16:06:39.148 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-05-08T16:06:39.148 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-05-08T16:06:39.148 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-05-08T16:06:39.148 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=f37874d1-6990-4767-8498-03a75cc31118 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/2 --osd-journal=td/test-erasure-code/2/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.48735/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-05-08T16:06:39.148 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-05-08T16:06:39.178 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:06:39.186+0000 7fc9464d67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:06:39.204 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:06:39.210+0000 7fc9464d67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:06:39.205 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:06:39.214+0000 7fc9464d67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:06:39.539 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-05-08T16:06:39.540 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-08T16:06:39.540 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-05-08T16:06:39.540 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-08T16:06:39.540 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-08T16:06:39.540 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:06:39.540 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-08T16:06:39.540 INFO:tasks.workunit.client.0.smithi172.stdout:0 2024-05-08T16:06:39.541 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:06:39.541 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T16:06:39.755 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:06:39.762+0000 7fc9464d67c0 -1 Falling back to public interface 2024-05-08T16:06:39.977 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:06:40.979 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:06:40.979 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:06:40.979 INFO:tasks.workunit.client.0.smithi172.stdout:1 2024-05-08T16:06:40.980 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-08T16:06:40.980 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:06:40.980 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T16:06:41.378 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:06:42.379 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:06:42.380 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:06:42.380 INFO:tasks.workunit.client.0.smithi172.stdout:2 2024-05-08T16:06:42.380 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-08T16:06:42.380 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:06:42.380 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T16:06:42.415 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:06:42.422+0000 7fc9464d67c0 -1 osd.2 0 log_to_monitors true 2024-05-08T16:06:42.827 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:06:43.828 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:06:43.828 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:06:43.829 INFO:tasks.workunit.client.0.smithi172.stdout:3 2024-05-08T16:06:43.829 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-05-08T16:06:43.829 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:06:43.829 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T16:06:44.361 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:06:45.363 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:06:45.363 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:06:45.363 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-05-08T16:06:45.364 INFO:tasks.workunit.client.0.smithi172.stdout:4 2024-05-08T16:06:45.364 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:06:45.364 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T16:06:45.858 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:06:46.860 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:06:46.860 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:06:46.860 INFO:tasks.workunit.client.0.smithi172.stdout:5 2024-05-08T16:06:46.861 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-05-08T16:06:46.861 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:06:46.861 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T16:06:47.340 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:06:48.342 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:06:48.342 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:06:48.342 INFO:tasks.workunit.client.0.smithi172.stdout:6 2024-05-08T16:06:48.343 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-05-08T16:06:48.343 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:06:48.344 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T16:06:48.835 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:06:49.837 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:06:49.838 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:06:49.838 INFO:tasks.workunit.client.0.smithi172.stdout:7 2024-05-08T16:06:49.838 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-05-08T16:06:49.839 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:06:49.839 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T16:06:50.384 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:06:51.385 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:06:51.386 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:06:51.386 INFO:tasks.workunit.client.0.smithi172.stdout:8 2024-05-08T16:06:51.386 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-05-08T16:06:51.387 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:06:51.387 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T16:06:51.876 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:06:52.878 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:06:52.878 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:06:52.878 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-05-08T16:06:52.878 INFO:tasks.workunit.client.0.smithi172.stdout:9 2024-05-08T16:06:52.879 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:06:52.879 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T16:06:53.376 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:06:54.378 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:06:54.378 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:06:54.378 INFO:tasks.workunit.client.0.smithi172.stdout:10 2024-05-08T16:06:54.379 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-05-08T16:06:54.379 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:06:54.379 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T16:06:54.818 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:06:55.820 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:06:55.820 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:06:55.820 INFO:tasks.workunit.client.0.smithi172.stdout:11 2024-05-08T16:06:55.821 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-05-08T16:06:55.821 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:06:55.821 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T16:06:56.366 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:06:57.368 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:06:57.368 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:06:57.368 INFO:tasks.workunit.client.0.smithi172.stdout:12 2024-05-08T16:06:57.369 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-05-08T16:06:57.369 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:06:57.369 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T16:06:58.396 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:06:59.398 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:06:59.398 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:06:59.398 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-05-08T16:06:59.398 INFO:tasks.workunit.client.0.smithi172.stdout:13 2024-05-08T16:06:59.399 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:06:59.399 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T16:07:00.224 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:07:01.226 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:07:01.226 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:07:01.226 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-05-08T16:07:01.226 INFO:tasks.workunit.client.0.smithi172.stdout:14 2024-05-08T16:07:01.226 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:07:01.227 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T16:07:01.987 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:07:02.990 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:07:02.990 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:07:02.990 INFO:tasks.workunit.client.0.smithi172.stdout:15 2024-05-08T16:07:02.991 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-05-08T16:07:02.991 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:07:02.991 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T16:07:04.130 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:07:05.133 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:07:05.133 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:07:05.134 INFO:tasks.workunit.client.0.smithi172.stdout:16 2024-05-08T16:07:05.134 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-05-08T16:07:05.135 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:07:05.135 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T16:07:05.960 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:07:06.962 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:07:06.962 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:07:06.962 INFO:tasks.workunit.client.0.smithi172.stdout:17 2024-05-08T16:07:06.963 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-05-08T16:07:06.963 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:07:06.963 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T16:07:07.629 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:07:08.630 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:07:08.631 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:07:08.631 INFO:tasks.workunit.client.0.smithi172.stdout:18 2024-05-08T16:07:08.631 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-05-08T16:07:08.633 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:07:08.633 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T16:07:09.069 INFO:tasks.workunit.client.0.smithi172.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/1917154460,v1:127.0.0.1:6819/1917154460] [v2:127.0.0.1:6820/1917154460,v1:127.0.0.1:6821/1917154460] exists,up 09f538fa-242a-496b-8a2c-83b2f8a35870 2024-05-08T16:07:09.073 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-08T16:07:09.073 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-08T16:07:09.073 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-08T16:07:09.073 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:36: run: for id in $(seq 0 10) 2024-05-08T16:07:09.073 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:37: run: run_osd td/test-erasure-code 3 2024-05-08T16:07:09.073 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-code 2024-05-08T16:07:09.073 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-05-08T16:07:09.073 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2024-05-08T16:07:09.073 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-05-08T16:07:09.073 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-code/3 2024-05-08T16:07:09.073 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=f37874d1-6990-4767-8498-03a75cc31118 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false' 2024-05-08T16:07:09.073 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-08T16:07:09.074 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-05-08T16:07:09.074 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-08T16:07:09.074 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-code/3' 2024-05-08T16:07:09.074 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code/3/journal' 2024-05-08T16:07:09.074 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-05-08T16:07:09.074 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-05-08T16:07:09.074 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-code' 2024-05-08T16:07:09.074 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-05-08T16:07:09.074 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-08T16:07:09.074 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-08T16:07:09.074 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-08T16:07:09.074 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T16:07:09.074 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.48735 2024-05-08T16:07:09.074 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.48735/$cluster-$name.asok' 2024-05-08T16:07:09.075 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.48735/$cluster-$name.asok' 2024-05-08T16:07:09.075 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-05-08T16:07:09.075 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-05-08T16:07:09.075 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-05-08T16:07:09.075 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-code/$name.log' 2024-05-08T16:07:09.075 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-code/$name.pid' 2024-05-08T16:07:09.075 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-08T16:07:09.075 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-08T16:07:09.075 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-08T16:07:09.075 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-08T16:07:09.075 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-05-08T16:07:09.075 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-05-08T16:07:09.075 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-code/3 2024-05-08T16:07:09.076 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-05-08T16:07:09.076 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=a0610a6e-a18c-4a81-9882-d0ee54a891b7 2024-05-08T16:07:09.076 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 a0610a6e-a18c-4a81-9882-d0ee54a891b7' 2024-05-08T16:07:09.076 INFO:tasks.workunit.client.0.smithi172.stdout:add osd3 a0610a6e-a18c-4a81-9882-d0ee54a891b7 2024-05-08T16:07:09.077 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-05-08T16:07:09.097 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAtoztmQaBLBhAA76TwC9hAS548pd8c1Sgpdg== 2024-05-08T16:07:09.097 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAtoztmQaBLBhAA76TwC9hAS548pd8c1Sgpdg=="}' 2024-05-08T16:07:09.097 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new a0610a6e-a18c-4a81-9882-d0ee54a891b7 -i td/test-erasure-code/3/new.json 2024-05-08T16:07:09.609 INFO:tasks.workunit.client.0.smithi172.stdout:3 2024-05-08T16:07:09.624 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-code/3/new.json 2024-05-08T16:07:09.626 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=f37874d1-6990-4767-8498-03a75cc31118 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/3 --osd-journal=td/test-erasure-code/3/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.48735/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAtoztmQaBLBhAA76TwC9hAS548pd8c1Sgpdg== --osd-uuid a0610a6e-a18c-4a81-9882-d0ee54a891b7 2024-05-08T16:07:09.659 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:07:09.666+0000 7f86f44cf7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:07:09.670 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:07:09.678+0000 7f86f44cf7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:07:09.671 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:07:09.678+0000 7f86f44cf7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:07:09.671 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:07:09.678+0000 7f86f44cf7c0 -1 bluestore(td/test-erasure-code/3/block) _read_bdev_label failed to open td/test-erasure-code/3/block: (2) No such file or directory 2024-05-08T16:07:09.671 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:07:09.678+0000 7f86f44cf7c0 -1 bluestore(td/test-erasure-code/3/block) _read_bdev_label failed to open td/test-erasure-code/3/block: (2) No such file or directory 2024-05-08T16:07:09.671 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:07:09.678+0000 7f86f44cf7c0 -1 bluestore(td/test-erasure-code/3/block) _read_bdev_label failed to open td/test-erasure-code/3/block: (2) No such file or directory 2024-05-08T16:07:09.709 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:07:09.718+0000 7f86f44cf7c0 -1 bluestore(td/test-erasure-code/3) _read_fsid unparsable uuid 2024-05-08T16:07:16.501 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-code/3/keyring 2024-05-08T16:07:16.502 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-05-08T16:07:16.503 INFO:tasks.workunit.client.0.smithi172.stdout:adding osd3 key to auth repository 2024-05-08T16:07:16.503 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2024-05-08T16:07:16.503 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-code/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-05-08T16:07:17.036 INFO:tasks.workunit.client.0.smithi172.stdout:start osd.3 2024-05-08T16:07:17.037 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2024-05-08T16:07:17.037 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-05-08T16:07:17.037 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-05-08T16:07:17.037 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=f37874d1-6990-4767-8498-03a75cc31118 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/3 --osd-journal=td/test-erasure-code/3/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.48735/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-05-08T16:07:17.037 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-05-08T16:07:17.067 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:07:17.074+0000 7f32a2b6f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:07:17.091 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:07:17.098+0000 7f32a2b6f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:07:17.092 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:07:17.098+0000 7f32a2b6f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:07:17.512 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2024-05-08T16:07:17.513 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-08T16:07:17.513 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-05-08T16:07:17.513 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-08T16:07:17.513 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-08T16:07:17.513 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:07:17.513 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-08T16:07:17.513 INFO:tasks.workunit.client.0.smithi172.stdout:0 2024-05-08T16:07:17.514 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:07:17.514 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T16:07:17.666 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:07:17.674+0000 7f32a2b6f7c0 -1 Falling back to public interface 2024-05-08T16:07:17.937 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:07:18.938 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:07:18.939 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:07:18.939 INFO:tasks.workunit.client.0.smithi172.stdout:1 2024-05-08T16:07:18.940 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-08T16:07:18.940 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:07:18.940 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T16:07:19.385 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:07:20.249 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:07:20.258+0000 7f32a2b6f7c0 -1 osd.3 0 log_to_monitors true 2024-05-08T16:07:20.387 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:07:20.387 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:07:20.388 INFO:tasks.workunit.client.0.smithi172.stdout:2 2024-05-08T16:07:20.388 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-08T16:07:20.388 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:07:20.388 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T16:07:20.972 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:07:21.974 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:07:21.974 INFO:tasks.workunit.client.0.smithi172.stdout:3 2024-05-08T16:07:21.975 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:07:21.975 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-05-08T16:07:21.975 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:07:21.975 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T16:07:22.458 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:07:23.459 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:07:23.460 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:07:23.460 INFO:tasks.workunit.client.0.smithi172.stdout:4 2024-05-08T16:07:23.460 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-05-08T16:07:23.460 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:07:23.461 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T16:07:23.887 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:07:24.888 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:07:24.888 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:07:24.889 INFO:tasks.workunit.client.0.smithi172.stdout:5 2024-05-08T16:07:24.889 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-05-08T16:07:24.889 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:07:24.889 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T16:07:25.438 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:07:26.438 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:07:26.438 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:07:26.438 INFO:tasks.workunit.client.0.smithi172.stdout:6 2024-05-08T16:07:26.439 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-05-08T16:07:26.439 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:07:26.439 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T16:07:26.883 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:07:27.885 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:07:27.885 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:07:27.885 INFO:tasks.workunit.client.0.smithi172.stdout:7 2024-05-08T16:07:27.886 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-05-08T16:07:27.886 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:07:27.886 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T16:07:28.463 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:07:29.465 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:07:29.465 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:07:29.465 INFO:tasks.workunit.client.0.smithi172.stdout:8 2024-05-08T16:07:29.466 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-05-08T16:07:29.466 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:07:29.466 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T16:07:29.904 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:07:30.906 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:07:30.906 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:07:30.906 INFO:tasks.workunit.client.0.smithi172.stdout:9 2024-05-08T16:07:30.907 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-05-08T16:07:30.907 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:07:30.907 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T16:07:31.452 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:07:32.454 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:07:32.454 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:07:32.455 INFO:tasks.workunit.client.0.smithi172.stdout:10 2024-05-08T16:07:32.455 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-05-08T16:07:32.455 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:07:32.455 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T16:07:32.880 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:07:33.882 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:07:33.882 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:07:33.883 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-05-08T16:07:33.883 INFO:tasks.workunit.client.0.smithi172.stdout:11 2024-05-08T16:07:33.883 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:07:33.883 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T16:07:34.598 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:07:35.600 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:07:35.600 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:07:35.600 INFO:tasks.workunit.client.0.smithi172.stdout:12 2024-05-08T16:07:35.601 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-05-08T16:07:35.601 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:07:35.601 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T16:07:36.556 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:07:37.558 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:07:37.558 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:07:37.558 INFO:tasks.workunit.client.0.smithi172.stdout:13 2024-05-08T16:07:37.559 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-05-08T16:07:37.559 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:07:37.559 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T16:07:38.414 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:07:39.415 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:07:39.416 INFO:tasks.workunit.client.0.smithi172.stdout:14 2024-05-08T16:07:39.416 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:07:39.416 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-05-08T16:07:39.416 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:07:39.417 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T16:07:40.274 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:07:41.276 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:07:41.277 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:07:41.277 INFO:tasks.workunit.client.0.smithi172.stdout:15 2024-05-08T16:07:41.277 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-05-08T16:07:41.277 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:07:41.277 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T16:07:42.129 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:07:43.131 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:07:43.131 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:07:43.131 INFO:tasks.workunit.client.0.smithi172.stdout:16 2024-05-08T16:07:43.132 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-05-08T16:07:43.132 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:07:43.132 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T16:07:43.868 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:07:44.870 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:07:44.870 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:07:44.870 INFO:tasks.workunit.client.0.smithi172.stdout:17 2024-05-08T16:07:44.871 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-05-08T16:07:44.871 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:07:44.871 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T16:07:45.347 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:07:46.349 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:07:46.349 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:07:46.349 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-05-08T16:07:46.350 INFO:tasks.workunit.client.0.smithi172.stdout:18 2024-05-08T16:07:46.350 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:07:46.350 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T16:07:46.792 INFO:tasks.workunit.client.0.smithi172.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/37179492,v1:127.0.0.1:6827/37179492] [v2:127.0.0.1:6828/37179492,v1:127.0.0.1:6829/37179492] exists,up a0610a6e-a18c-4a81-9882-d0ee54a891b7 2024-05-08T16:07:46.795 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-08T16:07:46.795 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-08T16:07:46.796 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-08T16:07:46.796 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:36: run: for id in $(seq 0 10) 2024-05-08T16:07:46.796 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:37: run: run_osd td/test-erasure-code 4 2024-05-08T16:07:46.796 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-code 2024-05-08T16:07:46.796 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-05-08T16:07:46.796 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=4 2024-05-08T16:07:46.796 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-05-08T16:07:46.796 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-code/4 2024-05-08T16:07:46.796 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=f37874d1-6990-4767-8498-03a75cc31118 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false' 2024-05-08T16:07:46.796 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-08T16:07:46.796 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-05-08T16:07:46.796 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-08T16:07:46.796 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-code/4' 2024-05-08T16:07:46.796 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code/4/journal' 2024-05-08T16:07:46.796 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-05-08T16:07:46.797 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-05-08T16:07:46.797 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-code' 2024-05-08T16:07:46.797 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-05-08T16:07:46.797 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-08T16:07:46.797 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-08T16:07:46.797 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-08T16:07:46.797 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T16:07:46.797 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.48735 2024-05-08T16:07:46.797 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.48735/$cluster-$name.asok' 2024-05-08T16:07:46.797 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.48735/$cluster-$name.asok' 2024-05-08T16:07:46.797 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-05-08T16:07:46.797 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-05-08T16:07:46.797 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-05-08T16:07:46.797 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-code/$name.log' 2024-05-08T16:07:46.798 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-code/$name.pid' 2024-05-08T16:07:46.798 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-08T16:07:46.798 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-08T16:07:46.798 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-08T16:07:46.798 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-08T16:07:46.798 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-05-08T16:07:46.798 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-05-08T16:07:46.798 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-code/4 2024-05-08T16:07:46.799 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-05-08T16:07:46.799 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=11dded9e-4d2a-4910-96cf-3e3c4f021d06 2024-05-08T16:07:46.799 INFO:tasks.workunit.client.0.smithi172.stdout:add osd4 11dded9e-4d2a-4910-96cf-3e3c4f021d06 2024-05-08T16:07:46.799 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd4 11dded9e-4d2a-4910-96cf-3e3c4f021d06' 2024-05-08T16:07:46.799 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-05-08T16:07:46.818 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBSoztmtw1ZMRAAz+BpQ3vqjnVTZCjDsQYwkQ== 2024-05-08T16:07:46.818 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBSoztmtw1ZMRAAz+BpQ3vqjnVTZCjDsQYwkQ=="}' 2024-05-08T16:07:46.819 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 11dded9e-4d2a-4910-96cf-3e3c4f021d06 -i td/test-erasure-code/4/new.json 2024-05-08T16:07:47.302 INFO:tasks.workunit.client.0.smithi172.stdout:4 2024-05-08T16:07:47.318 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-code/4/new.json 2024-05-08T16:07:47.319 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 4 --fsid=f37874d1-6990-4767-8498-03a75cc31118 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/4 --osd-journal=td/test-erasure-code/4/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.48735/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBSoztmtw1ZMRAAz+BpQ3vqjnVTZCjDsQYwkQ== --osd-uuid 11dded9e-4d2a-4910-96cf-3e3c4f021d06 2024-05-08T16:07:47.352 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:07:47.357+0000 7f98bd55f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:07:47.355 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:07:47.361+0000 7f98bd55f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:07:47.356 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:07:47.361+0000 7f98bd55f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:07:47.357 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:07:47.365+0000 7f98bd55f7c0 -1 bluestore(td/test-erasure-code/4/block) _read_bdev_label failed to open td/test-erasure-code/4/block: (2) No such file or directory 2024-05-08T16:07:47.357 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:07:47.365+0000 7f98bd55f7c0 -1 bluestore(td/test-erasure-code/4/block) _read_bdev_label failed to open td/test-erasure-code/4/block: (2) No such file or directory 2024-05-08T16:07:47.357 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:07:47.365+0000 7f98bd55f7c0 -1 bluestore(td/test-erasure-code/4/block) _read_bdev_label failed to open td/test-erasure-code/4/block: (2) No such file or directory 2024-05-08T16:07:47.387 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:07:47.393+0000 7f98bd55f7c0 -1 bluestore(td/test-erasure-code/4) _read_fsid unparsable uuid 2024-05-08T16:07:54.107 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-code/4/keyring 2024-05-08T16:07:54.107 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-05-08T16:07:54.108 INFO:tasks.workunit.client.0.smithi172.stdout:adding osd4 key to auth repository 2024-05-08T16:07:54.109 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd4 key to auth repository 2024-05-08T16:07:54.109 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-code/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-05-08T16:07:54.672 INFO:tasks.workunit.client.0.smithi172.stdout:start osd.4 2024-05-08T16:07:54.673 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.4 2024-05-08T16:07:54.673 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 4 --fsid=f37874d1-6990-4767-8498-03a75cc31118 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/4 --osd-journal=td/test-erasure-code/4/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.48735/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-05-08T16:07:54.673 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-05-08T16:07:54.673 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-05-08T16:07:54.673 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-05-08T16:07:54.704 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:07:54.709+0000 7f468dcd07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:07:54.706 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:07:54.713+0000 7f468dcd07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:07:54.707 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:07:54.713+0000 7f468dcd07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:07:55.124 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 4 2024-05-08T16:07:55.124 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-08T16:07:55.124 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=4 2024-05-08T16:07:55.124 INFO:tasks.workunit.client.0.smithi172.stdout:0 2024-05-08T16:07:55.125 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-08T16:07:55.125 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-08T16:07:55.125 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:07:55.125 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-08T16:07:55.125 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-08T16:07:55.125 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:07:55.254 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:07:55.261+0000 7f468dcd07c0 -1 Falling back to public interface 2024-05-08T16:07:55.660 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:07:56.661 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:07:56.661 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:07:56.662 INFO:tasks.workunit.client.0.smithi172.stdout:1 2024-05-08T16:07:56.662 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-08T16:07:56.662 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:07:56.662 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-08T16:07:57.076 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:07:57.885 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:07:57.893+0000 7f468dcd07c0 -1 osd.4 0 log_to_monitors true 2024-05-08T16:07:58.078 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:07:58.079 INFO:tasks.workunit.client.0.smithi172.stdout:2 2024-05-08T16:07:58.079 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:07:58.079 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-08T16:07:58.079 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:07:58.079 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-08T16:07:58.771 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:07:59.772 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:07:59.772 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:07:59.773 INFO:tasks.workunit.client.0.smithi172.stdout:3 2024-05-08T16:07:59.773 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-05-08T16:07:59.773 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:07:59.773 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-08T16:08:00.205 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:08:01.207 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:08:01.207 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:08:01.208 INFO:tasks.workunit.client.0.smithi172.stdout:4 2024-05-08T16:08:01.208 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-05-08T16:08:01.208 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:08:01.208 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-08T16:08:01.841 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:08:02.842 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:08:02.843 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:08:02.843 INFO:tasks.workunit.client.0.smithi172.stdout:5 2024-05-08T16:08:02.843 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-05-08T16:08:02.843 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:08:02.844 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-08T16:08:03.341 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:08:04.343 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:08:04.343 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:08:04.343 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-05-08T16:08:04.343 INFO:tasks.workunit.client.0.smithi172.stdout:6 2024-05-08T16:08:04.344 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:08:04.344 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-08T16:08:04.881 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:08:05.883 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:08:05.883 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:08:05.883 INFO:tasks.workunit.client.0.smithi172.stdout:7 2024-05-08T16:08:05.884 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-05-08T16:08:05.884 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:08:05.884 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-08T16:08:06.314 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:08:07.315 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:08:07.315 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:08:07.316 INFO:tasks.workunit.client.0.smithi172.stdout:8 2024-05-08T16:08:07.316 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-05-08T16:08:07.316 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:08:07.316 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-08T16:08:07.827 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:08:08.829 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:08:08.829 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:08:08.829 INFO:tasks.workunit.client.0.smithi172.stdout:9 2024-05-08T16:08:08.830 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-05-08T16:08:08.830 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:08:08.830 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-08T16:08:09.340 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:08:10.341 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:08:10.342 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:08:10.342 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-05-08T16:08:10.342 INFO:tasks.workunit.client.0.smithi172.stdout:10 2024-05-08T16:08:10.342 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:08:10.343 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-08T16:08:10.846 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:08:11.848 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:08:11.848 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:08:11.848 INFO:tasks.workunit.client.0.smithi172.stdout:11 2024-05-08T16:08:11.849 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-05-08T16:08:11.849 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:08:11.849 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-08T16:08:12.630 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:08:13.632 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:08:13.632 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:08:13.632 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-05-08T16:08:13.632 INFO:tasks.workunit.client.0.smithi172.stdout:12 2024-05-08T16:08:13.633 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:08:13.633 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-08T16:08:14.461 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:08:15.462 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:08:15.462 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:08:15.463 INFO:tasks.workunit.client.0.smithi172.stdout:13 2024-05-08T16:08:15.463 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-05-08T16:08:15.463 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:08:15.463 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-08T16:08:16.574 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:08:17.575 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:08:17.575 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:08:17.576 INFO:tasks.workunit.client.0.smithi172.stdout:14 2024-05-08T16:08:17.576 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-05-08T16:08:17.576 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:08:17.576 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-08T16:08:18.438 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:08:19.440 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:08:19.440 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:08:19.441 INFO:tasks.workunit.client.0.smithi172.stdout:15 2024-05-08T16:08:19.441 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-05-08T16:08:19.441 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:08:19.441 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-08T16:08:20.048 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:08:21.049 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:08:21.049 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:08:21.049 INFO:tasks.workunit.client.0.smithi172.stdout:16 2024-05-08T16:08:21.050 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-05-08T16:08:21.050 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:08:21.050 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-08T16:08:22.155 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:08:23.156 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:08:23.156 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:08:23.156 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-05-08T16:08:23.157 INFO:tasks.workunit.client.0.smithi172.stdout:17 2024-05-08T16:08:23.157 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:08:23.157 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-08T16:08:23.580 INFO:tasks.workunit.client.0.smithi172.stdout:osd.4 up in weight 1 up_from 25 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/2985329546,v1:127.0.0.1:6835/2985329546] [v2:127.0.0.1:6836/2985329546,v1:127.0.0.1:6837/2985329546] exists,up 11dded9e-4d2a-4910-96cf-3e3c4f021d06 2024-05-08T16:08:23.584 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-08T16:08:23.584 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-08T16:08:23.584 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-08T16:08:23.584 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:36: run: for id in $(seq 0 10) 2024-05-08T16:08:23.584 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:37: run: run_osd td/test-erasure-code 5 2024-05-08T16:08:23.584 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-code 2024-05-08T16:08:23.584 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-05-08T16:08:23.584 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=5 2024-05-08T16:08:23.585 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-05-08T16:08:23.585 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-code/5 2024-05-08T16:08:23.585 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=f37874d1-6990-4767-8498-03a75cc31118 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false' 2024-05-08T16:08:23.585 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-08T16:08:23.585 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-05-08T16:08:23.585 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-08T16:08:23.585 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-code/5' 2024-05-08T16:08:23.585 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code/5/journal' 2024-05-08T16:08:23.585 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-05-08T16:08:23.585 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-05-08T16:08:23.585 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-code' 2024-05-08T16:08:23.585 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-05-08T16:08:23.585 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-08T16:08:23.585 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-08T16:08:23.585 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-08T16:08:23.586 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T16:08:23.586 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.48735 2024-05-08T16:08:23.586 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.48735/$cluster-$name.asok' 2024-05-08T16:08:23.586 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.48735/$cluster-$name.asok' 2024-05-08T16:08:23.586 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-05-08T16:08:23.586 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-05-08T16:08:23.586 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-05-08T16:08:23.586 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-code/$name.log' 2024-05-08T16:08:23.586 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-code/$name.pid' 2024-05-08T16:08:23.586 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-08T16:08:23.586 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-08T16:08:23.586 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-08T16:08:23.586 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-08T16:08:23.586 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-05-08T16:08:23.586 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-05-08T16:08:23.587 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-code/5 2024-05-08T16:08:23.587 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-05-08T16:08:23.587 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=608e9ac9-5b3f-4c5a-aa1f-92dc47044545 2024-05-08T16:08:23.587 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd5 608e9ac9-5b3f-4c5a-aa1f-92dc47044545' 2024-05-08T16:08:23.588 INFO:tasks.workunit.client.0.smithi172.stdout:add osd5 608e9ac9-5b3f-4c5a-aa1f-92dc47044545 2024-05-08T16:08:23.588 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-05-08T16:08:23.607 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQB3oztm3Sq9JBAAShFCQG7sQarFHiWSoEoFjA== 2024-05-08T16:08:23.607 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQB3oztm3Sq9JBAAShFCQG7sQarFHiWSoEoFjA=="}' 2024-05-08T16:08:23.607 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 608e9ac9-5b3f-4c5a-aa1f-92dc47044545 -i td/test-erasure-code/5/new.json 2024-05-08T16:08:24.314 INFO:tasks.workunit.client.0.smithi172.stdout:5 2024-05-08T16:08:24.336 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-code/5/new.json 2024-05-08T16:08:24.337 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 5 --fsid=f37874d1-6990-4767-8498-03a75cc31118 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/5 --osd-journal=td/test-erasure-code/5/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.48735/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQB3oztm3Sq9JBAAShFCQG7sQarFHiWSoEoFjA== --osd-uuid 608e9ac9-5b3f-4c5a-aa1f-92dc47044545 2024-05-08T16:08:24.371 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:08:24.377+0000 7fb6bce277c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:08:24.373 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:08:24.381+0000 7fb6bce277c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:08:24.374 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:08:24.381+0000 7fb6bce277c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:08:24.375 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:08:24.381+0000 7fb6bce277c0 -1 bluestore(td/test-erasure-code/5/block) _read_bdev_label failed to open td/test-erasure-code/5/block: (2) No such file or directory 2024-05-08T16:08:24.375 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:08:24.381+0000 7fb6bce277c0 -1 bluestore(td/test-erasure-code/5/block) _read_bdev_label failed to open td/test-erasure-code/5/block: (2) No such file or directory 2024-05-08T16:08:24.375 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:08:24.381+0000 7fb6bce277c0 -1 bluestore(td/test-erasure-code/5/block) _read_bdev_label failed to open td/test-erasure-code/5/block: (2) No such file or directory 2024-05-08T16:08:24.496 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:08:24.501+0000 7fb6bce277c0 -1 bluestore(td/test-erasure-code/5) _read_fsid unparsable uuid 2024-05-08T16:08:31.473 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-code/5/keyring 2024-05-08T16:08:31.473 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-05-08T16:08:31.473 INFO:tasks.workunit.client.0.smithi172.stdout:adding osd5 key to auth repository 2024-05-08T16:08:31.474 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd5 key to auth repository 2024-05-08T16:08:31.474 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-code/5/keyring auth add osd.5 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-05-08T16:08:32.031 INFO:tasks.workunit.client.0.smithi172.stdout:start osd.5 2024-05-08T16:08:32.032 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.5 2024-05-08T16:08:32.032 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 5 --fsid=f37874d1-6990-4767-8498-03a75cc31118 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/5 --osd-journal=td/test-erasure-code/5/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.48735/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-05-08T16:08:32.032 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-05-08T16:08:32.032 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-05-08T16:08:32.033 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-05-08T16:08:32.062 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:08:32.069+0000 7f2a4e5607c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:08:32.072 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:08:32.077+0000 7f2a4e5607c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:08:32.073 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:08:32.081+0000 7f2a4e5607c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:08:32.426 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 5 2024-05-08T16:08:32.426 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-08T16:08:32.426 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=5 2024-05-08T16:08:32.426 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-08T16:08:32.426 INFO:tasks.workunit.client.0.smithi172.stdout:0 2024-05-08T16:08:32.427 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-08T16:08:32.427 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:08:32.427 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-08T16:08:32.427 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:08:32.427 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-05-08T16:08:32.642 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:08:32.649+0000 7f2a4e5607c0 -1 Falling back to public interface 2024-05-08T16:08:32.843 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:08:33.845 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:08:33.845 INFO:tasks.workunit.client.0.smithi172.stdout:1 2024-05-08T16:08:33.845 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:08:33.846 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-08T16:08:33.846 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:08:33.846 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-05-08T16:08:34.245 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:08:35.247 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:08:35.247 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:08:35.248 INFO:tasks.workunit.client.0.smithi172.stdout:2 2024-05-08T16:08:35.248 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-08T16:08:35.248 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:08:35.248 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-05-08T16:08:35.360 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:08:35.365+0000 7f2a4e5607c0 -1 osd.5 0 log_to_monitors true 2024-05-08T16:08:35.706 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:08:36.708 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:08:36.708 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:08:36.708 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-05-08T16:08:36.708 INFO:tasks.workunit.client.0.smithi172.stdout:3 2024-05-08T16:08:36.708 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:08:36.708 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-05-08T16:08:37.203 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:08:38.205 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:08:38.205 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:08:38.205 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-05-08T16:08:38.205 INFO:tasks.workunit.client.0.smithi172.stdout:4 2024-05-08T16:08:38.206 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:08:38.206 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-05-08T16:08:38.651 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:08:39.653 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:08:39.653 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:08:39.653 INFO:tasks.workunit.client.0.smithi172.stdout:5 2024-05-08T16:08:39.654 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-05-08T16:08:39.654 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:08:39.654 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-05-08T16:08:40.088 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:08:41.089 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:08:41.089 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:08:41.090 INFO:tasks.workunit.client.0.smithi172.stdout:6 2024-05-08T16:08:41.090 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-05-08T16:08:41.090 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:08:41.090 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-05-08T16:08:41.538 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:08:42.540 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:08:42.540 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:08:42.540 INFO:tasks.workunit.client.0.smithi172.stdout:7 2024-05-08T16:08:42.541 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-05-08T16:08:42.541 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:08:42.541 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-05-08T16:08:42.984 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:08:43.986 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:08:43.986 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:08:43.986 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-05-08T16:08:43.986 INFO:tasks.workunit.client.0.smithi172.stdout:8 2024-05-08T16:08:43.987 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:08:43.987 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-05-08T16:08:44.621 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:08:45.623 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:08:45.623 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:08:45.623 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-05-08T16:08:45.623 INFO:tasks.workunit.client.0.smithi172.stdout:9 2024-05-08T16:08:45.624 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:08:45.624 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-05-08T16:08:46.030 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:08:47.032 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:08:47.032 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:08:47.032 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-05-08T16:08:47.032 INFO:tasks.workunit.client.0.smithi172.stdout:10 2024-05-08T16:08:47.033 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:08:47.033 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-05-08T16:08:47.515 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:08:48.517 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:08:48.517 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:08:48.517 INFO:tasks.workunit.client.0.smithi172.stdout:11 2024-05-08T16:08:48.518 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-05-08T16:08:48.518 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:08:48.518 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-05-08T16:08:48.925 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:08:49.927 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:08:49.927 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:08:49.927 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-05-08T16:08:49.928 INFO:tasks.workunit.client.0.smithi172.stdout:12 2024-05-08T16:08:49.928 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:08:49.928 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-05-08T16:08:51.048 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:08:52.050 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:08:52.050 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:08:52.050 INFO:tasks.workunit.client.0.smithi172.stdout:13 2024-05-08T16:08:52.051 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-05-08T16:08:52.051 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:08:52.051 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-05-08T16:08:52.985 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:08:53.986 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:08:53.987 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:08:53.987 INFO:tasks.workunit.client.0.smithi172.stdout:14 2024-05-08T16:08:53.987 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-05-08T16:08:53.987 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:08:53.988 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-05-08T16:08:54.813 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:08:55.815 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:08:55.815 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:08:55.815 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-05-08T16:08:55.816 INFO:tasks.workunit.client.0.smithi172.stdout:15 2024-05-08T16:08:55.816 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:08:55.816 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-05-08T16:08:56.774 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:08:57.775 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:08:57.776 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:08:57.776 INFO:tasks.workunit.client.0.smithi172.stdout:16 2024-05-08T16:08:57.776 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-05-08T16:08:57.776 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:08:57.776 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-05-08T16:08:59.071 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:09:00.073 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:09:00.073 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:09:00.073 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-05-08T16:09:00.073 INFO:tasks.workunit.client.0.smithi172.stdout:17 2024-05-08T16:09:00.074 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:09:00.074 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-05-08T16:09:00.914 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:09:01.916 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:09:01.916 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:09:01.916 INFO:tasks.workunit.client.0.smithi172.stdout:18 2024-05-08T16:09:01.917 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-05-08T16:09:01.917 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:09:01.917 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-05-08T16:09:02.340 INFO:tasks.workunit.client.0.smithi172.stdout:osd.5 up in weight 1 up_from 30 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6842/3705378844,v1:127.0.0.1:6843/3705378844] [v2:127.0.0.1:6844/3705378844,v1:127.0.0.1:6845/3705378844] exists,up 608e9ac9-5b3f-4c5a-aa1f-92dc47044545 2024-05-08T16:09:02.344 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-08T16:09:02.344 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-08T16:09:02.344 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-08T16:09:02.344 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:36: run: for id in $(seq 0 10) 2024-05-08T16:09:02.345 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:37: run: run_osd td/test-erasure-code 6 2024-05-08T16:09:02.345 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-code 2024-05-08T16:09:02.345 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-05-08T16:09:02.345 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=6 2024-05-08T16:09:02.345 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-05-08T16:09:02.345 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-code/6 2024-05-08T16:09:02.345 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=f37874d1-6990-4767-8498-03a75cc31118 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false' 2024-05-08T16:09:02.345 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-08T16:09:02.345 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-05-08T16:09:02.345 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-08T16:09:02.345 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-code/6' 2024-05-08T16:09:02.345 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code/6/journal' 2024-05-08T16:09:02.345 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-05-08T16:09:02.345 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-05-08T16:09:02.346 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-code' 2024-05-08T16:09:02.346 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-05-08T16:09:02.346 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-08T16:09:02.346 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-08T16:09:02.346 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-08T16:09:02.346 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T16:09:02.346 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.48735 2024-05-08T16:09:02.346 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.48735/$cluster-$name.asok' 2024-05-08T16:09:02.346 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.48735/$cluster-$name.asok' 2024-05-08T16:09:02.346 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-05-08T16:09:02.346 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-05-08T16:09:02.346 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-05-08T16:09:02.346 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-code/$name.log' 2024-05-08T16:09:02.346 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-code/$name.pid' 2024-05-08T16:09:02.347 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-08T16:09:02.347 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-08T16:09:02.347 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-08T16:09:02.347 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-08T16:09:02.347 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-05-08T16:09:02.347 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-05-08T16:09:02.347 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-code/6 2024-05-08T16:09:02.348 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-05-08T16:09:02.348 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=eea3e6ba-2b1f-4c9e-99f4-8290d432b42b 2024-05-08T16:09:02.348 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd6 eea3e6ba-2b1f-4c9e-99f4-8290d432b42b' 2024-05-08T16:09:02.348 INFO:tasks.workunit.client.0.smithi172.stdout:add osd6 eea3e6ba-2b1f-4c9e-99f4-8290d432b42b 2024-05-08T16:09:02.348 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-05-08T16:09:02.368 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCeoztmZYKGFhAAqfl701th8SGbf0l12sya7w== 2024-05-08T16:09:02.368 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCeoztmZYKGFhAAqfl701th8SGbf0l12sya7w=="}' 2024-05-08T16:09:02.368 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new eea3e6ba-2b1f-4c9e-99f4-8290d432b42b -i td/test-erasure-code/6/new.json 2024-05-08T16:09:02.874 INFO:tasks.workunit.client.0.smithi172.stdout:6 2024-05-08T16:09:02.890 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-code/6/new.json 2024-05-08T16:09:02.891 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 6 --fsid=f37874d1-6990-4767-8498-03a75cc31118 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/6 --osd-journal=td/test-erasure-code/6/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.48735/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCeoztmZYKGFhAAqfl701th8SGbf0l12sya7w== --osd-uuid eea3e6ba-2b1f-4c9e-99f4-8290d432b42b 2024-05-08T16:09:02.924 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:09:02.932+0000 7f90cc1a47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:09:02.941 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:09:02.948+0000 7f90cc1a47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:09:02.942 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:09:02.948+0000 7f90cc1a47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:09:02.943 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:09:02.948+0000 7f90cc1a47c0 -1 bluestore(td/test-erasure-code/6/block) _read_bdev_label failed to open td/test-erasure-code/6/block: (2) No such file or directory 2024-05-08T16:09:02.943 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:09:02.948+0000 7f90cc1a47c0 -1 bluestore(td/test-erasure-code/6/block) _read_bdev_label failed to open td/test-erasure-code/6/block: (2) No such file or directory 2024-05-08T16:09:02.943 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:09:02.948+0000 7f90cc1a47c0 -1 bluestore(td/test-erasure-code/6/block) _read_bdev_label failed to open td/test-erasure-code/6/block: (2) No such file or directory 2024-05-08T16:09:02.980 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:09:02.988+0000 7f90cc1a47c0 -1 bluestore(td/test-erasure-code/6) _read_fsid unparsable uuid 2024-05-08T16:09:10.050 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-code/6/keyring 2024-05-08T16:09:10.051 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-05-08T16:09:10.052 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd6 key to auth repository 2024-05-08T16:09:10.052 INFO:tasks.workunit.client.0.smithi172.stdout:adding osd6 key to auth repository 2024-05-08T16:09:10.052 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-code/6/keyring auth add osd.6 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-05-08T16:09:10.616 INFO:tasks.workunit.client.0.smithi172.stdout:start osd.6 2024-05-08T16:09:10.617 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.6 2024-05-08T16:09:10.617 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 6 --fsid=f37874d1-6990-4767-8498-03a75cc31118 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/6 --osd-journal=td/test-erasure-code/6/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.48735/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-05-08T16:09:10.617 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-05-08T16:09:10.617 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-05-08T16:09:10.617 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-05-08T16:09:10.647 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:09:10.652+0000 7f23389e07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:09:10.666 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:09:10.672+0000 7f23389e07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:09:10.667 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:09:10.672+0000 7f23389e07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:09:11.054 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 6 2024-05-08T16:09:11.054 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-08T16:09:11.054 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=6 2024-05-08T16:09:11.054 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-08T16:09:11.054 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-08T16:09:11.054 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:09:11.054 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-08T16:09:11.054 INFO:tasks.workunit.client.0.smithi172.stdout:0 2024-05-08T16:09:11.055 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:09:11.055 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-05-08T16:09:11.237 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:09:11.244+0000 7f23389e07c0 -1 Falling back to public interface 2024-05-08T16:09:11.476 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:09:12.477 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:09:12.477 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:09:12.477 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-08T16:09:12.478 INFO:tasks.workunit.client.0.smithi172.stdout:1 2024-05-08T16:09:12.478 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:09:12.478 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-05-08T16:09:12.892 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:09:13.893 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:09:13.894 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:09:13.894 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-08T16:09:13.894 INFO:tasks.workunit.client.0.smithi172.stdout:2 2024-05-08T16:09:13.894 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:09:13.895 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-05-08T16:09:13.900 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:09:13.904+0000 7f23389e07c0 -1 osd.6 0 log_to_monitors true 2024-05-08T16:09:14.453 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:09:15.455 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:09:15.455 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:09:15.455 INFO:tasks.workunit.client.0.smithi172.stdout:3 2024-05-08T16:09:15.456 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-05-08T16:09:15.456 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:09:15.457 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-05-08T16:09:15.935 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:09:16.936 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:09:16.936 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:09:16.937 INFO:tasks.workunit.client.0.smithi172.stdout:4 2024-05-08T16:09:16.937 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-05-08T16:09:16.937 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:09:16.937 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-05-08T16:09:17.544 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:09:18.547 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:09:18.547 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:09:18.548 INFO:tasks.workunit.client.0.smithi172.stdout:5 2024-05-08T16:09:18.548 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-05-08T16:09:18.548 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:09:18.548 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-05-08T16:09:19.097 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:09:20.099 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:09:20.099 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:09:20.099 INFO:tasks.workunit.client.0.smithi172.stdout:6 2024-05-08T16:09:20.100 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-05-08T16:09:20.100 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:09:20.101 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-05-08T16:09:20.519 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:09:21.520 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:09:21.521 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:09:21.521 INFO:tasks.workunit.client.0.smithi172.stdout:7 2024-05-08T16:09:21.521 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-05-08T16:09:21.521 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:09:21.521 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-05-08T16:09:21.941 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:09:22.943 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:09:22.943 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:09:22.944 INFO:tasks.workunit.client.0.smithi172.stdout:8 2024-05-08T16:09:22.944 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-05-08T16:09:22.944 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:09:22.944 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-05-08T16:09:23.445 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:09:24.446 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:09:24.447 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:09:24.447 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-05-08T16:09:24.447 INFO:tasks.workunit.client.0.smithi172.stdout:9 2024-05-08T16:09:24.447 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:09:24.447 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-05-08T16:09:25.027 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:09:26.029 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:09:26.029 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:09:26.029 INFO:tasks.workunit.client.0.smithi172.stdout:10 2024-05-08T16:09:26.030 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-05-08T16:09:26.030 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:09:26.030 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-05-08T16:09:26.461 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:09:27.462 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:09:27.463 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:09:27.463 INFO:tasks.workunit.client.0.smithi172.stdout:11 2024-05-08T16:09:27.463 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-05-08T16:09:27.463 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:09:27.463 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-05-08T16:09:27.981 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:09:28.983 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:09:28.984 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:09:28.984 INFO:tasks.workunit.client.0.smithi172.stdout:12 2024-05-08T16:09:28.984 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-05-08T16:09:28.985 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:09:28.985 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-05-08T16:09:29.830 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:09:30.832 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:09:30.832 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:09:30.833 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-05-08T16:09:30.833 INFO:tasks.workunit.client.0.smithi172.stdout:13 2024-05-08T16:09:30.833 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:09:30.833 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-05-08T16:09:32.151 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:09:33.153 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:09:33.153 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:09:33.153 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-05-08T16:09:33.154 INFO:tasks.workunit.client.0.smithi172.stdout:14 2024-05-08T16:09:33.154 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:09:33.154 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-05-08T16:09:33.879 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:09:34.880 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:09:34.881 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:09:34.881 INFO:tasks.workunit.client.0.smithi172.stdout:15 2024-05-08T16:09:34.881 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-05-08T16:09:34.882 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:09:34.882 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-05-08T16:09:36.059 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:09:37.061 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:09:37.061 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:09:37.061 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-05-08T16:09:37.061 INFO:tasks.workunit.client.0.smithi172.stdout:16 2024-05-08T16:09:37.062 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:09:37.062 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-05-08T16:09:37.940 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:09:38.942 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:09:38.942 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:09:38.942 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-05-08T16:09:38.942 INFO:tasks.workunit.client.0.smithi172.stdout:17 2024-05-08T16:09:38.943 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:09:38.943 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-05-08T16:09:39.421 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:09:40.423 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:09:40.423 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:09:40.423 INFO:tasks.workunit.client.0.smithi172.stdout:18 2024-05-08T16:09:40.424 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-05-08T16:09:40.424 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:09:40.424 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-05-08T16:09:40.966 INFO:tasks.workunit.client.0.smithi172.stdout:osd.6 up in weight 1 up_from 35 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6850/598852430,v1:127.0.0.1:6851/598852430] [v2:127.0.0.1:6852/598852430,v1:127.0.0.1:6853/598852430] exists,up eea3e6ba-2b1f-4c9e-99f4-8290d432b42b 2024-05-08T16:09:40.970 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-08T16:09:40.970 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-08T16:09:40.970 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-08T16:09:40.970 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:36: run: for id in $(seq 0 10) 2024-05-08T16:09:40.970 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:37: run: run_osd td/test-erasure-code 7 2024-05-08T16:09:40.970 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-code 2024-05-08T16:09:40.970 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-05-08T16:09:40.970 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=7 2024-05-08T16:09:40.970 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-05-08T16:09:40.970 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-code/7 2024-05-08T16:09:40.970 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=f37874d1-6990-4767-8498-03a75cc31118 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false' 2024-05-08T16:09:40.971 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-08T16:09:40.971 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-05-08T16:09:40.971 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-08T16:09:40.971 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-code/7' 2024-05-08T16:09:40.971 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code/7/journal' 2024-05-08T16:09:40.971 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-05-08T16:09:40.971 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-05-08T16:09:40.971 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-code' 2024-05-08T16:09:40.971 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-05-08T16:09:40.971 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-08T16:09:40.971 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-08T16:09:40.971 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-08T16:09:40.971 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T16:09:40.971 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.48735 2024-05-08T16:09:40.971 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.48735/$cluster-$name.asok' 2024-05-08T16:09:40.972 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.48735/$cluster-$name.asok' 2024-05-08T16:09:40.972 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-05-08T16:09:40.972 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-05-08T16:09:40.972 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-05-08T16:09:40.972 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-code/$name.log' 2024-05-08T16:09:40.972 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-code/$name.pid' 2024-05-08T16:09:40.972 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-08T16:09:40.972 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-08T16:09:40.972 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-08T16:09:40.972 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-08T16:09:40.972 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-05-08T16:09:40.972 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-05-08T16:09:40.972 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-code/7 2024-05-08T16:09:40.973 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-05-08T16:09:40.973 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=f43933bb-426b-47b9-b0b2-4b43e3e3e4b8 2024-05-08T16:09:40.973 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd7 f43933bb-426b-47b9-b0b2-4b43e3e3e4b8' 2024-05-08T16:09:40.973 INFO:tasks.workunit.client.0.smithi172.stdout:add osd7 f43933bb-426b-47b9-b0b2-4b43e3e3e4b8 2024-05-08T16:09:40.974 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-05-08T16:09:40.994 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDFoztmRtxIABAADzreAVhh1rOd/wPZOHFbwA== 2024-05-08T16:09:40.995 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDFoztmRtxIABAADzreAVhh1rOd/wPZOHFbwA=="}' 2024-05-08T16:09:40.995 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new f43933bb-426b-47b9-b0b2-4b43e3e3e4b8 -i td/test-erasure-code/7/new.json 2024-05-08T16:09:41.549 INFO:tasks.workunit.client.0.smithi172.stdout:7 2024-05-08T16:09:41.565 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-code/7/new.json 2024-05-08T16:09:41.569 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 7 --fsid=f37874d1-6990-4767-8498-03a75cc31118 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/7 --osd-journal=td/test-erasure-code/7/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.48735/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDFoztmRtxIABAADzreAVhh1rOd/wPZOHFbwA== --osd-uuid f43933bb-426b-47b9-b0b2-4b43e3e3e4b8 2024-05-08T16:09:41.599 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:09:41.604+0000 7f1dea4707c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:09:41.635 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:09:41.640+0000 7f1dea4707c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:09:41.636 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:09:41.644+0000 7f1dea4707c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:09:41.636 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:09:41.644+0000 7f1dea4707c0 -1 bluestore(td/test-erasure-code/7/block) _read_bdev_label failed to open td/test-erasure-code/7/block: (2) No such file or directory 2024-05-08T16:09:41.636 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:09:41.644+0000 7f1dea4707c0 -1 bluestore(td/test-erasure-code/7/block) _read_bdev_label failed to open td/test-erasure-code/7/block: (2) No such file or directory 2024-05-08T16:09:41.637 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:09:41.644+0000 7f1dea4707c0 -1 bluestore(td/test-erasure-code/7/block) _read_bdev_label failed to open td/test-erasure-code/7/block: (2) No such file or directory 2024-05-08T16:09:41.675 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:09:41.680+0000 7f1dea4707c0 -1 bluestore(td/test-erasure-code/7) _read_fsid unparsable uuid 2024-05-08T16:09:49.182 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-code/7/keyring 2024-05-08T16:09:49.182 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-05-08T16:09:49.183 INFO:tasks.workunit.client.0.smithi172.stdout:adding osd7 key to auth repository 2024-05-08T16:09:49.183 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd7 key to auth repository 2024-05-08T16:09:49.183 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-code/7/keyring auth add osd.7 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-05-08T16:09:49.739 INFO:tasks.workunit.client.0.smithi172.stdout:start osd.7 2024-05-08T16:09:49.739 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.7 2024-05-08T16:09:49.740 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 7 --fsid=f37874d1-6990-4767-8498-03a75cc31118 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/7 --osd-journal=td/test-erasure-code/7/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.48735/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-05-08T16:09:49.740 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-05-08T16:09:49.740 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-05-08T16:09:49.740 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-05-08T16:09:49.769 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:09:49.776+0000 7fb2f12797c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:09:49.771 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:09:49.776+0000 7fb2f12797c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:09:49.773 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:09:49.780+0000 7fb2f12797c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:09:50.140 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 7 2024-05-08T16:09:50.141 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-08T16:09:50.141 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=7 2024-05-08T16:09:50.141 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-08T16:09:50.141 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-08T16:09:50.141 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:09:50.141 INFO:tasks.workunit.client.0.smithi172.stdout:0 2024-05-08T16:09:50.142 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-08T16:09:50.142 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:09:50.142 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-05-08T16:09:50.345 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:09:50.352+0000 7fb2f12797c0 -1 Falling back to public interface 2024-05-08T16:09:50.617 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:09:51.619 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:09:51.619 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:09:51.620 INFO:tasks.workunit.client.0.smithi172.stdout:1 2024-05-08T16:09:51.620 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-08T16:09:51.620 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:09:51.620 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-05-08T16:09:52.021 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:09:53.023 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:09:53.023 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:09:53.023 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-08T16:09:53.023 INFO:tasks.workunit.client.0.smithi172.stdout:2 2024-05-08T16:09:53.024 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:09:53.024 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-05-08T16:09:53.179 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:09:53.184+0000 7fb2f12797c0 -1 osd.7 0 log_to_monitors true 2024-05-08T16:09:53.535 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:09:54.537 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:09:54.538 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:09:54.538 INFO:tasks.workunit.client.0.smithi172.stdout:3 2024-05-08T16:09:54.538 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-05-08T16:09:54.539 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:09:54.539 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-05-08T16:09:55.025 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:09:56.027 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:09:56.027 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:09:56.027 INFO:tasks.workunit.client.0.smithi172.stdout:4 2024-05-08T16:09:56.028 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-05-08T16:09:56.028 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:09:56.028 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-05-08T16:09:56.670 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:09:57.671 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:09:57.672 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:09:57.672 INFO:tasks.workunit.client.0.smithi172.stdout:5 2024-05-08T16:09:57.672 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-05-08T16:09:57.672 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:09:57.672 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-05-08T16:09:58.107 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:09:59.109 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:09:59.109 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:09:59.109 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-05-08T16:09:59.110 INFO:tasks.workunit.client.0.smithi172.stdout:6 2024-05-08T16:09:59.110 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:09:59.110 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-05-08T16:09:59.592 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:10:00.593 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:10:00.593 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:10:00.594 INFO:tasks.workunit.client.0.smithi172.stdout:7 2024-05-08T16:10:00.594 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-05-08T16:10:00.594 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:10:00.594 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-05-08T16:10:01.043 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:10:02.045 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:10:02.045 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:10:02.045 INFO:tasks.workunit.client.0.smithi172.stdout:8 2024-05-08T16:10:02.046 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-05-08T16:10:02.046 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:10:02.046 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-05-08T16:10:02.646 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:10:03.648 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:10:03.648 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:10:03.648 INFO:tasks.workunit.client.0.smithi172.stdout:9 2024-05-08T16:10:03.648 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-05-08T16:10:03.649 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:10:03.649 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-05-08T16:10:04.118 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:10:05.119 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:10:05.120 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:10:05.120 INFO:tasks.workunit.client.0.smithi172.stdout:10 2024-05-08T16:10:05.121 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-05-08T16:10:05.121 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:10:05.121 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-05-08T16:10:05.552 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:10:06.558 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:10:06.558 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:10:06.558 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-05-08T16:10:06.559 INFO:tasks.workunit.client.0.smithi172.stdout:11 2024-05-08T16:10:06.559 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:10:06.560 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-05-08T16:10:06.994 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:10:07.996 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:10:07.996 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:10:07.996 INFO:tasks.workunit.client.0.smithi172.stdout:12 2024-05-08T16:10:07.997 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-05-08T16:10:07.997 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:10:07.997 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-05-08T16:10:08.975 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:10:09.976 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:10:09.977 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:10:09.977 INFO:tasks.workunit.client.0.smithi172.stdout:13 2024-05-08T16:10:09.977 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-05-08T16:10:09.977 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:10:09.978 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-05-08T16:10:10.796 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:10:11.798 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:10:11.799 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:10:11.799 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-05-08T16:10:11.799 INFO:tasks.workunit.client.0.smithi172.stdout:14 2024-05-08T16:10:11.799 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:10:11.799 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-05-08T16:10:12.658 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:10:13.659 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:10:13.660 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:10:13.660 INFO:tasks.workunit.client.0.smithi172.stdout:15 2024-05-08T16:10:13.661 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-05-08T16:10:13.661 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:10:13.661 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-05-08T16:10:14.334 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:10:15.336 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:10:15.336 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:10:15.336 INFO:tasks.workunit.client.0.smithi172.stdout:16 2024-05-08T16:10:15.337 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-05-08T16:10:15.337 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:10:15.337 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-05-08T16:10:15.997 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:10:16.998 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:10:16.999 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:10:16.999 INFO:tasks.workunit.client.0.smithi172.stdout:17 2024-05-08T16:10:16.999 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-05-08T16:10:17.000 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:10:17.000 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-05-08T16:10:17.996 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:10:18.997 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:10:18.997 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:10:18.998 INFO:tasks.workunit.client.0.smithi172.stdout:18 2024-05-08T16:10:18.998 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-05-08T16:10:18.998 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:10:18.998 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-05-08T16:10:19.853 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:10:20.854 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:10:20.854 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:10:20.855 INFO:tasks.workunit.client.0.smithi172.stdout:19 2024-05-08T16:10:20.855 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-05-08T16:10:20.855 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:10:20.855 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-05-08T16:10:21.273 INFO:tasks.workunit.client.0.smithi172.stdout:osd.7 up in weight 1 up_from 40 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6858/2270887273,v1:127.0.0.1:6859/2270887273] [v2:127.0.0.1:6860/2270887273,v1:127.0.0.1:6861/2270887273] exists,up f43933bb-426b-47b9-b0b2-4b43e3e3e4b8 2024-05-08T16:10:21.275 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-08T16:10:21.275 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-08T16:10:21.275 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-08T16:10:21.275 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:36: run: for id in $(seq 0 10) 2024-05-08T16:10:21.275 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:37: run: run_osd td/test-erasure-code 8 2024-05-08T16:10:21.275 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-code 2024-05-08T16:10:21.275 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-05-08T16:10:21.275 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=8 2024-05-08T16:10:21.275 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-05-08T16:10:21.275 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-code/8 2024-05-08T16:10:21.276 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=f37874d1-6990-4767-8498-03a75cc31118 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false' 2024-05-08T16:10:21.276 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-08T16:10:21.276 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-05-08T16:10:21.276 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-08T16:10:21.276 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-code/8' 2024-05-08T16:10:21.276 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code/8/journal' 2024-05-08T16:10:21.276 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-05-08T16:10:21.276 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-05-08T16:10:21.276 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-code' 2024-05-08T16:10:21.276 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-05-08T16:10:21.278 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-08T16:10:21.278 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-08T16:10:21.279 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-08T16:10:21.279 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T16:10:21.279 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.48735 2024-05-08T16:10:21.279 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.48735/$cluster-$name.asok' 2024-05-08T16:10:21.279 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.48735/$cluster-$name.asok' 2024-05-08T16:10:21.279 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-05-08T16:10:21.279 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-05-08T16:10:21.279 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-05-08T16:10:21.279 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-code/$name.log' 2024-05-08T16:10:21.279 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-code/$name.pid' 2024-05-08T16:10:21.279 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-08T16:10:21.279 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-08T16:10:21.279 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-08T16:10:21.279 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-08T16:10:21.280 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-05-08T16:10:21.280 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-05-08T16:10:21.280 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-code/8 2024-05-08T16:10:21.280 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-05-08T16:10:21.280 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=507f961e-4490-40b7-bbc8-e4c0f8b3bdcd 2024-05-08T16:10:21.280 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd8 507f961e-4490-40b7-bbc8-e4c0f8b3bdcd' 2024-05-08T16:10:21.281 INFO:tasks.workunit.client.0.smithi172.stdout:add osd8 507f961e-4490-40b7-bbc8-e4c0f8b3bdcd 2024-05-08T16:10:21.281 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-05-08T16:10:21.296 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDtoztmvT9XEhAAetWzopKMyaMMr/dLTxB80w== 2024-05-08T16:10:21.296 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDtoztmvT9XEhAAetWzopKMyaMMr/dLTxB80w=="}' 2024-05-08T16:10:21.297 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 507f961e-4490-40b7-bbc8-e4c0f8b3bdcd -i td/test-erasure-code/8/new.json 2024-05-08T16:10:21.935 INFO:tasks.workunit.client.0.smithi172.stdout:8 2024-05-08T16:10:21.950 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-code/8/new.json 2024-05-08T16:10:21.952 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 8 --fsid=f37874d1-6990-4767-8498-03a75cc31118 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/8 --osd-journal=td/test-erasure-code/8/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.48735/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDtoztmvT9XEhAAetWzopKMyaMMr/dLTxB80w== --osd-uuid 507f961e-4490-40b7-bbc8-e4c0f8b3bdcd 2024-05-08T16:10:21.984 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:10:21.992+0000 7fbebce187c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:10:22.022 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:10:22.028+0000 7fbebce187c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:10:22.024 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:10:22.032+0000 7fbebce187c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:10:22.024 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:10:22.032+0000 7fbebce187c0 -1 bluestore(td/test-erasure-code/8/block) _read_bdev_label failed to open td/test-erasure-code/8/block: (2) No such file or directory 2024-05-08T16:10:22.024 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:10:22.032+0000 7fbebce187c0 -1 bluestore(td/test-erasure-code/8/block) _read_bdev_label failed to open td/test-erasure-code/8/block: (2) No such file or directory 2024-05-08T16:10:22.024 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:10:22.032+0000 7fbebce187c0 -1 bluestore(td/test-erasure-code/8/block) _read_bdev_label failed to open td/test-erasure-code/8/block: (2) No such file or directory 2024-05-08T16:10:22.069 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:10:22.076+0000 7fbebce187c0 -1 bluestore(td/test-erasure-code/8) _read_fsid unparsable uuid 2024-05-08T16:10:29.240 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-code/8/keyring 2024-05-08T16:10:29.240 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-05-08T16:10:29.241 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd8 key to auth repository 2024-05-08T16:10:29.242 INFO:tasks.workunit.client.0.smithi172.stdout:adding osd8 key to auth repository 2024-05-08T16:10:29.242 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-code/8/keyring auth add osd.8 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-05-08T16:10:29.793 INFO:tasks.workunit.client.0.smithi172.stdout:start osd.8 2024-05-08T16:10:29.793 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.8 2024-05-08T16:10:29.793 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 8 --fsid=f37874d1-6990-4767-8498-03a75cc31118 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/8 --osd-journal=td/test-erasure-code/8/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.48735/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-05-08T16:10:29.794 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-05-08T16:10:29.794 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-05-08T16:10:29.794 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-05-08T16:10:29.823 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:10:29.831+0000 7fc7683677c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:10:29.825 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:10:29.831+0000 7fc7683677c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:10:29.826 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:10:29.831+0000 7fc7683677c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:10:30.203 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 8 2024-05-08T16:10:30.203 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-08T16:10:30.204 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=8 2024-05-08T16:10:30.204 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-08T16:10:30.204 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-08T16:10:30.204 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:10:30.204 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-08T16:10:30.204 INFO:tasks.workunit.client.0.smithi172.stdout:0 2024-05-08T16:10:30.204 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:10:30.205 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-05-08T16:10:30.386 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:10:30.391+0000 7fc7683677c0 -1 Falling back to public interface 2024-05-08T16:10:30.653 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:10:31.654 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:10:31.654 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:10:31.654 INFO:tasks.workunit.client.0.smithi172.stdout:1 2024-05-08T16:10:31.655 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-08T16:10:31.655 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:10:31.655 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-05-08T16:10:32.054 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:10:33.056 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:10:33.056 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:10:33.056 INFO:tasks.workunit.client.0.smithi172.stdout:2 2024-05-08T16:10:33.057 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-08T16:10:33.057 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:10:33.057 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-05-08T16:10:33.188 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:10:33.195+0000 7fc7683677c0 -1 osd.8 0 log_to_monitors true 2024-05-08T16:10:33.577 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:10:34.579 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:10:34.579 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:10:34.580 INFO:tasks.workunit.client.0.smithi172.stdout:3 2024-05-08T16:10:34.580 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-05-08T16:10:34.580 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:10:34.580 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-05-08T16:10:35.132 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:10:36.134 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:10:36.134 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:10:36.134 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-05-08T16:10:36.134 INFO:tasks.workunit.client.0.smithi172.stdout:4 2024-05-08T16:10:36.135 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:10:36.135 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-05-08T16:10:36.745 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:10:37.747 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:10:37.747 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:10:37.747 INFO:tasks.workunit.client.0.smithi172.stdout:5 2024-05-08T16:10:37.748 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-05-08T16:10:37.748 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:10:37.748 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-05-08T16:10:38.314 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:10:39.315 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:10:39.316 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:10:39.316 INFO:tasks.workunit.client.0.smithi172.stdout:6 2024-05-08T16:10:39.316 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-05-08T16:10:39.316 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:10:39.317 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-05-08T16:10:39.764 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:10:40.765 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:10:40.766 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:10:40.766 INFO:tasks.workunit.client.0.smithi172.stdout:7 2024-05-08T16:10:40.766 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-05-08T16:10:40.767 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:10:40.767 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-05-08T16:10:41.413 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:10:42.415 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:10:42.415 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:10:42.415 INFO:tasks.workunit.client.0.smithi172.stdout:8 2024-05-08T16:10:42.416 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-05-08T16:10:42.416 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:10:42.416 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-05-08T16:10:42.886 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:10:43.887 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:10:43.887 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:10:43.888 INFO:tasks.workunit.client.0.smithi172.stdout:9 2024-05-08T16:10:43.888 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-05-08T16:10:43.888 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:10:43.888 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-05-08T16:10:44.460 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:10:45.462 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:10:45.462 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:10:45.463 INFO:tasks.workunit.client.0.smithi172.stdout:10 2024-05-08T16:10:45.463 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-05-08T16:10:45.463 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:10:45.463 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-05-08T16:10:45.935 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:10:46.937 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:10:46.937 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:10:46.937 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-05-08T16:10:46.937 INFO:tasks.workunit.client.0.smithi172.stdout:11 2024-05-08T16:10:46.938 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:10:46.938 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-05-08T16:10:47.524 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:10:48.526 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:10:48.526 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:10:48.527 INFO:tasks.workunit.client.0.smithi172.stdout:12 2024-05-08T16:10:48.527 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-05-08T16:10:48.527 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:10:48.527 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-05-08T16:10:49.484 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:10:50.487 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:10:50.487 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:10:50.487 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-05-08T16:10:50.488 INFO:tasks.workunit.client.0.smithi172.stdout:13 2024-05-08T16:10:50.488 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:10:50.489 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-05-08T16:10:51.316 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:10:52.318 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:10:52.318 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:10:52.318 INFO:tasks.workunit.client.0.smithi172.stdout:14 2024-05-08T16:10:52.319 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-05-08T16:10:52.319 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:10:52.319 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-05-08T16:10:53.655 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:10:54.657 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:10:54.657 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:10:54.657 INFO:tasks.workunit.client.0.smithi172.stdout:15 2024-05-08T16:10:54.658 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-05-08T16:10:54.658 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:10:54.658 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-05-08T16:10:55.468 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:10:56.470 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:10:56.470 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:10:56.470 INFO:tasks.workunit.client.0.smithi172.stdout:16 2024-05-08T16:10:56.471 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-05-08T16:10:56.471 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:10:56.471 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-05-08T16:10:57.661 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:10:58.663 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:10:58.663 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:10:58.663 INFO:tasks.workunit.client.0.smithi172.stdout:17 2024-05-08T16:10:58.664 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-05-08T16:10:58.664 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:10:58.664 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-05-08T16:10:59.679 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:11:00.681 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:11:00.681 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:11:00.681 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-05-08T16:11:00.682 INFO:tasks.workunit.client.0.smithi172.stdout:18 2024-05-08T16:11:00.682 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:11:00.682 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-05-08T16:11:01.097 INFO:tasks.workunit.client.0.smithi172.stdout:osd.8 up in weight 1 up_from 45 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6866/4119555052,v1:127.0.0.1:6867/4119555052] [v2:127.0.0.1:6868/4119555052,v1:127.0.0.1:6869/4119555052] exists,up 507f961e-4490-40b7-bbc8-e4c0f8b3bdcd 2024-05-08T16:11:01.102 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-08T16:11:01.102 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-08T16:11:01.102 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-08T16:11:01.102 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:36: run: for id in $(seq 0 10) 2024-05-08T16:11:01.102 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:37: run: run_osd td/test-erasure-code 9 2024-05-08T16:11:01.102 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-code 2024-05-08T16:11:01.102 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-05-08T16:11:01.102 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=9 2024-05-08T16:11:01.102 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-05-08T16:11:01.102 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-code/9 2024-05-08T16:11:01.102 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=f37874d1-6990-4767-8498-03a75cc31118 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false' 2024-05-08T16:11:01.102 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-08T16:11:01.103 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-05-08T16:11:01.103 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-08T16:11:01.103 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-code/9' 2024-05-08T16:11:01.103 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code/9/journal' 2024-05-08T16:11:01.103 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-05-08T16:11:01.103 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-05-08T16:11:01.103 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-code' 2024-05-08T16:11:01.103 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-05-08T16:11:01.103 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-08T16:11:01.103 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-08T16:11:01.103 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-08T16:11:01.103 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T16:11:01.104 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.48735 2024-05-08T16:11:01.104 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.48735/$cluster-$name.asok' 2024-05-08T16:11:01.104 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.48735/$cluster-$name.asok' 2024-05-08T16:11:01.104 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-05-08T16:11:01.104 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-05-08T16:11:01.104 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-05-08T16:11:01.104 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-code/$name.log' 2024-05-08T16:11:01.104 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-code/$name.pid' 2024-05-08T16:11:01.104 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-08T16:11:01.104 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-08T16:11:01.104 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-08T16:11:01.104 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-08T16:11:01.104 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-05-08T16:11:01.105 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-05-08T16:11:01.105 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-code/9 2024-05-08T16:11:01.105 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-05-08T16:11:01.105 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=54514676-7fc6-4e04-8a2c-b3fc55239502 2024-05-08T16:11:01.105 INFO:tasks.workunit.client.0.smithi172.stdout:add osd9 54514676-7fc6-4e04-8a2c-b3fc55239502 2024-05-08T16:11:01.106 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd9 54514676-7fc6-4e04-8a2c-b3fc55239502' 2024-05-08T16:11:01.106 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-05-08T16:11:01.125 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAVpDtm4K4fCBAA2w2GkjqmN3QkU8TnUJyxvQ== 2024-05-08T16:11:01.126 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAVpDtm4K4fCBAA2w2GkjqmN3QkU8TnUJyxvQ=="}' 2024-05-08T16:11:01.126 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 54514676-7fc6-4e04-8a2c-b3fc55239502 -i td/test-erasure-code/9/new.json 2024-05-08T16:11:01.779 INFO:tasks.workunit.client.0.smithi172.stdout:9 2024-05-08T16:11:01.794 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-code/9/new.json 2024-05-08T16:11:01.795 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 9 --fsid=f37874d1-6990-4767-8498-03a75cc31118 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/9 --osd-journal=td/test-erasure-code/9/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.48735/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAVpDtm4K4fCBAA2w2GkjqmN3QkU8TnUJyxvQ== --osd-uuid 54514676-7fc6-4e04-8a2c-b3fc55239502 2024-05-08T16:11:01.828 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:11:01.839+0000 7f769a08b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:11:01.865 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:11:01.875+0000 7f769a08b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:11:01.866 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:11:01.875+0000 7f769a08b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:11:01.867 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:11:01.875+0000 7f769a08b7c0 -1 bluestore(td/test-erasure-code/9/block) _read_bdev_label failed to open td/test-erasure-code/9/block: (2) No such file or directory 2024-05-08T16:11:01.867 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:11:01.875+0000 7f769a08b7c0 -1 bluestore(td/test-erasure-code/9/block) _read_bdev_label failed to open td/test-erasure-code/9/block: (2) No such file or directory 2024-05-08T16:11:01.867 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:11:01.875+0000 7f769a08b7c0 -1 bluestore(td/test-erasure-code/9/block) _read_bdev_label failed to open td/test-erasure-code/9/block: (2) No such file or directory 2024-05-08T16:11:01.904 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:11:01.915+0000 7f769a08b7c0 -1 bluestore(td/test-erasure-code/9) _read_fsid unparsable uuid 2024-05-08T16:11:09.173 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-code/9/keyring 2024-05-08T16:11:09.173 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-05-08T16:11:09.174 INFO:tasks.workunit.client.0.smithi172.stdout:adding osd9 key to auth repository 2024-05-08T16:11:09.175 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd9 key to auth repository 2024-05-08T16:11:09.175 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-code/9/keyring auth add osd.9 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-05-08T16:11:09.734 INFO:tasks.workunit.client.0.smithi172.stdout:start osd.9 2024-05-08T16:11:09.735 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.9 2024-05-08T16:11:09.735 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 9 --fsid=f37874d1-6990-4767-8498-03a75cc31118 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/9 --osd-journal=td/test-erasure-code/9/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.48735/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-05-08T16:11:09.735 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-05-08T16:11:09.735 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-05-08T16:11:09.735 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-05-08T16:11:09.765 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:11:09.775+0000 7fc24c0e27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:11:09.805 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:11:09.815+0000 7fc24c0e27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:11:09.807 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:11:09.815+0000 7fc24c0e27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:11:10.194 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 9 2024-05-08T16:11:10.194 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-08T16:11:10.194 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=9 2024-05-08T16:11:10.194 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-08T16:11:10.194 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-08T16:11:10.195 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:11:10.195 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-08T16:11:10.195 INFO:tasks.workunit.client.0.smithi172.stdout:0 2024-05-08T16:11:10.196 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:11:10.196 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-05-08T16:11:10.380 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:11:10.390+0000 7fc24c0e27c0 -1 Falling back to public interface 2024-05-08T16:11:10.609 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:11:11.610 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:11:11.611 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:11:11.611 INFO:tasks.workunit.client.0.smithi172.stdout:1 2024-05-08T16:11:11.611 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-08T16:11:11.611 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:11:11.612 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-05-08T16:11:12.011 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:11:13.013 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:11:13.013 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:11:13.013 INFO:tasks.workunit.client.0.smithi172.stdout:2 2024-05-08T16:11:13.014 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-08T16:11:13.014 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:11:13.014 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-05-08T16:11:13.190 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:11:13.198+0000 7fc24c0e27c0 -1 osd.9 0 log_to_monitors true 2024-05-08T16:11:13.535 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:11:14.536 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:11:14.537 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:11:14.537 INFO:tasks.workunit.client.0.smithi172.stdout:3 2024-05-08T16:11:14.538 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-05-08T16:11:14.538 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:11:14.538 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-05-08T16:11:14.966 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:11:15.967 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:11:15.968 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:11:15.968 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-05-08T16:11:15.968 INFO:tasks.workunit.client.0.smithi172.stdout:4 2024-05-08T16:11:15.969 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:11:15.969 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-05-08T16:11:16.550 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:11:17.552 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:11:17.552 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:11:17.552 INFO:tasks.workunit.client.0.smithi172.stdout:5 2024-05-08T16:11:17.552 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-05-08T16:11:17.553 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:11:17.553 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-05-08T16:11:18.002 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:11:19.004 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:11:19.004 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:11:19.004 INFO:tasks.workunit.client.0.smithi172.stdout:6 2024-05-08T16:11:19.004 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-05-08T16:11:19.005 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:11:19.005 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-05-08T16:11:19.616 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:11:20.618 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:11:20.618 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:11:20.618 INFO:tasks.workunit.client.0.smithi172.stdout:7 2024-05-08T16:11:20.619 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-05-08T16:11:20.619 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:11:20.619 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-05-08T16:11:21.038 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:11:22.039 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:11:22.040 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:11:22.040 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-05-08T16:11:22.040 INFO:tasks.workunit.client.0.smithi172.stdout:8 2024-05-08T16:11:22.040 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:11:22.040 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-05-08T16:11:22.532 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:11:23.534 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:11:23.534 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:11:23.534 INFO:tasks.workunit.client.0.smithi172.stdout:9 2024-05-08T16:11:23.535 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-05-08T16:11:23.535 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:11:23.535 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-05-08T16:11:24.001 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:11:25.003 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:11:25.003 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:11:25.003 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-05-08T16:11:25.003 INFO:tasks.workunit.client.0.smithi172.stdout:10 2024-05-08T16:11:25.004 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:11:25.004 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-05-08T16:11:25.522 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:11:26.525 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:11:26.525 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:11:26.526 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-05-08T16:11:26.526 INFO:tasks.workunit.client.0.smithi172.stdout:11 2024-05-08T16:11:26.526 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:11:26.526 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-05-08T16:11:26.960 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:11:27.961 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:11:27.961 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:11:27.962 INFO:tasks.workunit.client.0.smithi172.stdout:12 2024-05-08T16:11:27.962 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-05-08T16:11:27.962 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:11:27.962 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-05-08T16:11:28.890 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:11:29.893 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:11:29.893 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:11:29.893 INFO:tasks.workunit.client.0.smithi172.stdout:13 2024-05-08T16:11:29.893 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-05-08T16:11:29.894 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:11:29.894 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-05-08T16:11:30.856 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:11:31.857 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:11:31.858 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:11:31.858 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-05-08T16:11:31.858 INFO:tasks.workunit.client.0.smithi172.stdout:14 2024-05-08T16:11:31.858 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:11:31.858 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-05-08T16:11:32.727 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:11:33.729 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:11:33.729 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:11:33.729 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-05-08T16:11:33.730 INFO:tasks.workunit.client.0.smithi172.stdout:15 2024-05-08T16:11:33.730 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:11:33.730 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-05-08T16:11:34.618 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:11:35.619 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:11:35.620 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:11:35.620 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-05-08T16:11:35.620 INFO:tasks.workunit.client.0.smithi172.stdout:16 2024-05-08T16:11:35.620 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:11:35.620 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-05-08T16:11:36.744 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:11:37.746 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:11:37.746 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:11:37.747 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-05-08T16:11:37.747 INFO:tasks.workunit.client.0.smithi172.stdout:17 2024-05-08T16:11:37.748 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:11:37.748 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-05-08T16:11:38.582 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:11:39.583 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:11:39.584 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:11:39.584 INFO:tasks.workunit.client.0.smithi172.stdout:18 2024-05-08T16:11:39.584 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-05-08T16:11:39.584 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:11:39.584 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-05-08T16:11:40.015 INFO:tasks.workunit.client.0.smithi172.stdout:osd.9 up in weight 1 up_from 50 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6874/592637954,v1:127.0.0.1:6875/592637954] [v2:127.0.0.1:6876/592637954,v1:127.0.0.1:6877/592637954] exists,up 54514676-7fc6-4e04-8a2c-b3fc55239502 2024-05-08T16:11:40.019 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-08T16:11:40.019 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-08T16:11:40.019 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-08T16:11:40.019 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:36: run: for id in $(seq 0 10) 2024-05-08T16:11:40.019 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:37: run: run_osd td/test-erasure-code 10 2024-05-08T16:11:40.019 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-code 2024-05-08T16:11:40.019 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-05-08T16:11:40.019 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=10 2024-05-08T16:11:40.019 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-05-08T16:11:40.020 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-code/10 2024-05-08T16:11:40.020 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=f37874d1-6990-4767-8498-03a75cc31118 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false' 2024-05-08T16:11:40.020 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-08T16:11:40.020 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-05-08T16:11:40.020 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-08T16:11:40.020 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-code/10' 2024-05-08T16:11:40.020 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-code/10/journal' 2024-05-08T16:11:40.020 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-05-08T16:11:40.020 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-05-08T16:11:40.020 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-code' 2024-05-08T16:11:40.020 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-05-08T16:11:40.020 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-08T16:11:40.020 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-08T16:11:40.020 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-08T16:11:40.021 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T16:11:40.021 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.48735 2024-05-08T16:11:40.021 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.48735/$cluster-$name.asok' 2024-05-08T16:11:40.021 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.48735/$cluster-$name.asok' 2024-05-08T16:11:40.021 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-05-08T16:11:40.021 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-05-08T16:11:40.021 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-05-08T16:11:40.021 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-code/$name.log' 2024-05-08T16:11:40.021 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-code/$name.pid' 2024-05-08T16:11:40.021 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-08T16:11:40.021 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-08T16:11:40.021 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-08T16:11:40.021 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-08T16:11:40.021 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-05-08T16:11:40.021 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-05-08T16:11:40.022 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-code/10 2024-05-08T16:11:40.022 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-05-08T16:11:40.022 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=1f4eaf7f-e79b-44c9-b620-dc9e74fde10d 2024-05-08T16:11:40.022 INFO:tasks.workunit.client.0.smithi172.stdout:add osd10 1f4eaf7f-e79b-44c9-b620-dc9e74fde10d 2024-05-08T16:11:40.023 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd10 1f4eaf7f-e79b-44c9-b620-dc9e74fde10d' 2024-05-08T16:11:40.023 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-05-08T16:11:40.041 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQA8pDtm/xgeAxAA6VIKwK2Sy0AKg6X031+JKg== 2024-05-08T16:11:40.041 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQA8pDtm/xgeAxAA6VIKwK2Sy0AKg6X031+JKg=="}' 2024-05-08T16:11:40.042 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 1f4eaf7f-e79b-44c9-b620-dc9e74fde10d -i td/test-erasure-code/10/new.json 2024-05-08T16:11:40.743 INFO:tasks.workunit.client.0.smithi172.stdout:10 2024-05-08T16:11:40.759 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-code/10/new.json 2024-05-08T16:11:40.760 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 10 --fsid=f37874d1-6990-4767-8498-03a75cc31118 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/10 --osd-journal=td/test-erasure-code/10/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.48735/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQA8pDtm/xgeAxAA6VIKwK2Sy0AKg6X031+JKg== --osd-uuid 1f4eaf7f-e79b-44c9-b620-dc9e74fde10d 2024-05-08T16:11:40.793 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:11:40.802+0000 7f5b2349e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:11:40.795 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:11:40.806+0000 7f5b2349e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:11:40.796 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:11:40.806+0000 7f5b2349e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:11:40.797 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:11:40.806+0000 7f5b2349e7c0 -1 bluestore(td/test-erasure-code/10/block) _read_bdev_label failed to open td/test-erasure-code/10/block: (2) No such file or directory 2024-05-08T16:11:40.797 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:11:40.806+0000 7f5b2349e7c0 -1 bluestore(td/test-erasure-code/10/block) _read_bdev_label failed to open td/test-erasure-code/10/block: (2) No such file or directory 2024-05-08T16:11:40.797 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:11:40.806+0000 7f5b2349e7c0 -1 bluestore(td/test-erasure-code/10/block) _read_bdev_label failed to open td/test-erasure-code/10/block: (2) No such file or directory 2024-05-08T16:11:40.973 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:11:40.982+0000 7f5b2349e7c0 -1 bluestore(td/test-erasure-code/10) _read_fsid unparsable uuid 2024-05-08T16:11:47.117 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-code/10/keyring 2024-05-08T16:11:47.117 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-05-08T16:11:47.118 INFO:tasks.workunit.client.0.smithi172.stdout:adding osd10 key to auth repository 2024-05-08T16:11:47.119 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd10 key to auth repository 2024-05-08T16:11:47.119 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-code/10/keyring auth add osd.10 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-05-08T16:11:47.673 INFO:tasks.workunit.client.0.smithi172.stdout:start osd.10 2024-05-08T16:11:47.674 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.10 2024-05-08T16:11:47.674 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 10 --fsid=f37874d1-6990-4767-8498-03a75cc31118 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/10 --osd-journal=td/test-erasure-code/10/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.48735/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-05-08T16:11:47.674 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-05-08T16:11:47.674 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-05-08T16:11:47.675 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-05-08T16:11:47.704 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:11:47.714+0000 7f74882e97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:11:47.746 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:11:47.754+0000 7f74882e97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:11:47.749 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:11:47.758+0000 7f74882e97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:11:48.125 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 10 2024-05-08T16:11:48.125 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-08T16:11:48.125 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=10 2024-05-08T16:11:48.126 INFO:tasks.workunit.client.0.smithi172.stdout:0 2024-05-08T16:11:48.126 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-08T16:11:48.126 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-08T16:11:48.126 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:11:48.126 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-08T16:11:48.127 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:11:48.127 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.10 up' 2024-05-08T16:11:48.309 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:11:48.318+0000 7f74882e97c0 -1 Falling back to public interface 2024-05-08T16:11:48.518 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:11:49.519 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:11:49.520 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:11:49.520 INFO:tasks.workunit.client.0.smithi172.stdout:1 2024-05-08T16:11:49.520 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-08T16:11:49.520 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:11:49.521 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.10 up' 2024-05-08T16:11:49.919 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:11:50.892 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:11:50.902+0000 7f74882e97c0 -1 osd.10 0 log_to_monitors true 2024-05-08T16:11:50.920 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:11:50.920 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:11:50.920 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-08T16:11:50.920 INFO:tasks.workunit.client.0.smithi172.stdout:2 2024-05-08T16:11:50.921 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:11:50.921 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.10 up' 2024-05-08T16:11:51.582 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:11:52.584 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:11:52.584 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:11:52.584 INFO:tasks.workunit.client.0.smithi172.stdout:3 2024-05-08T16:11:52.584 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-05-08T16:11:52.585 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:11:52.585 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.10 up' 2024-05-08T16:11:53.071 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:11:54.073 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:11:54.216 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:11:54.217 INFO:tasks.workunit.client.0.smithi172.stdout:4 2024-05-08T16:11:54.218 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-05-08T16:11:54.218 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:11:54.218 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.10 up' 2024-05-08T16:11:54.553 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:11:55.554 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:11:55.555 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:11:55.555 INFO:tasks.workunit.client.0.smithi172.stdout:5 2024-05-08T16:11:55.556 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-05-08T16:11:55.556 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:11:55.556 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.10 up' 2024-05-08T16:11:56.029 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:11:57.030 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:11:57.031 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:11:57.031 INFO:tasks.workunit.client.0.smithi172.stdout:6 2024-05-08T16:11:57.031 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-05-08T16:11:57.031 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:11:57.031 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.10 up' 2024-05-08T16:11:57.468 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:11:58.470 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:11:58.470 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:11:58.470 INFO:tasks.workunit.client.0.smithi172.stdout:7 2024-05-08T16:11:58.471 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-05-08T16:11:58.471 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:11:58.471 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.10 up' 2024-05-08T16:11:58.962 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:11:59.964 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:11:59.964 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:11:59.964 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-05-08T16:11:59.964 INFO:tasks.workunit.client.0.smithi172.stdout:8 2024-05-08T16:11:59.965 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:11:59.965 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.10 up' 2024-05-08T16:12:00.402 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:12:01.404 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:12:01.404 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:12:01.404 INFO:tasks.workunit.client.0.smithi172.stdout:9 2024-05-08T16:12:01.405 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-05-08T16:12:01.405 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:12:01.405 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.10 up' 2024-05-08T16:12:01.900 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:12:02.902 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:12:02.902 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:12:02.903 INFO:tasks.workunit.client.0.smithi172.stdout:10 2024-05-08T16:12:02.903 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-05-08T16:12:02.903 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:12:02.903 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.10 up' 2024-05-08T16:12:03.347 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:12:04.348 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:12:04.349 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:12:04.349 INFO:tasks.workunit.client.0.smithi172.stdout:11 2024-05-08T16:12:04.349 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-05-08T16:12:04.350 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:12:04.350 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.10 up' 2024-05-08T16:12:04.843 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:12:05.844 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:12:05.844 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:12:05.845 INFO:tasks.workunit.client.0.smithi172.stdout:12 2024-05-08T16:12:05.845 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-05-08T16:12:05.845 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:12:05.845 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.10 up' 2024-05-08T16:12:06.762 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:12:07.764 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:12:07.764 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:12:07.764 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-05-08T16:12:07.764 INFO:tasks.workunit.client.0.smithi172.stdout:13 2024-05-08T16:12:07.765 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:12:07.765 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.10 up' 2024-05-08T16:12:08.764 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:12:09.766 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:12:09.766 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:12:09.766 INFO:tasks.workunit.client.0.smithi172.stdout:14 2024-05-08T16:12:09.767 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-05-08T16:12:09.767 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:12:09.767 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.10 up' 2024-05-08T16:12:10.698 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:12:11.700 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:12:11.700 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:12:11.700 INFO:tasks.workunit.client.0.smithi172.stdout:15 2024-05-08T16:12:11.701 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-05-08T16:12:11.701 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:12:11.701 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.10 up' 2024-05-08T16:12:12.781 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:12:13.783 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:12:13.784 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:12:13.784 INFO:tasks.workunit.client.0.smithi172.stdout:16 2024-05-08T16:12:13.784 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-05-08T16:12:13.784 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:12:13.785 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.10 up' 2024-05-08T16:12:14.718 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:12:15.719 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:12:15.720 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:12:15.720 INFO:tasks.workunit.client.0.smithi172.stdout:17 2024-05-08T16:12:15.720 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-05-08T16:12:15.720 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:12:15.721 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.10 up' 2024-05-08T16:12:16.813 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:12:17.815 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:12:17.815 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:12:17.815 INFO:tasks.workunit.client.0.smithi172.stdout:18 2024-05-08T16:12:17.815 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-05-08T16:12:17.815 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:12:17.816 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.10 up' 2024-05-08T16:12:18.243 INFO:tasks.workunit.client.0.smithi172.stdout:osd.10 up in weight 1 up_from 55 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6882/4105237100,v1:127.0.0.1:6883/4105237100] [v2:127.0.0.1:6884/4105237100,v1:127.0.0.1:6885/4105237100] exists,up 1f4eaf7f-e79b-44c9-b620-dc9e74fde10d 2024-05-08T16:12:18.244 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-08T16:12:18.244 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-08T16:12:18.244 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-08T16:12:18.244 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:39: run: create_rbd_pool 2024-05-08T16:12:18.244 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-05-08T16:12:18.647 INFO:tasks.workunit.client.0.smithi172.stderr:pool 'rbd' does not exist 2024-05-08T16:12:18.663 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-05-08T16:12:18.663 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-05-08T16:12:19.296 INFO:tasks.workunit.client.0.smithi172.stderr:pool 'rbd' already exists 2024-05-08T16:12:19.315 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-05-08T16:12:20.316 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-05-08T16:12:23.636 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:40: run: wait_for_clean 2024-05-08T16:12:23.636 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-05-08T16:12:23.637 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-05-08T16:12:23.637 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-05-08T16:12:23.637 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-05-08T16:12:23.638 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-05-08T16:12:23.638 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-05-08T16:12:23.639 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-05-08T16:12:23.639 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-05-08T16:12:23.639 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-05-08T16:12:23.757 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-05-08T16:12:23.757 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-05-08T16:12:23.757 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-05-08T16:12:23.757 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-05-08T16:12:23.757 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-05-08T16:12:23.757 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-05-08T16:12:24.176 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-05-08T16:12:24.177 INFO:tasks.workunit.client.0.smithi172.stderr:1 2024-05-08T16:12:24.177 INFO:tasks.workunit.client.0.smithi172.stderr:2 2024-05-08T16:12:24.177 INFO:tasks.workunit.client.0.smithi172.stderr:3 2024-05-08T16:12:24.177 INFO:tasks.workunit.client.0.smithi172.stderr:4 2024-05-08T16:12:24.177 INFO:tasks.workunit.client.0.smithi172.stderr:5 2024-05-08T16:12:24.177 INFO:tasks.workunit.client.0.smithi172.stderr:6 2024-05-08T16:12:24.177 INFO:tasks.workunit.client.0.smithi172.stderr:7 2024-05-08T16:12:24.177 INFO:tasks.workunit.client.0.smithi172.stderr:8 2024-05-08T16:12:24.177 INFO:tasks.workunit.client.0.smithi172.stderr:9 2024-05-08T16:12:24.177 INFO:tasks.workunit.client.0.smithi172.stderr:10' 2024-05-08T16:12:24.177 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-05-08T16:12:24.177 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T16:12:24.178 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-05-08T16:12:24.386 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836565 2024-05-08T16:12:24.386 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836565 2024-05-08T16:12:24.386 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836565' 2024-05-08T16:12:24.386 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T16:12:24.387 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-05-08T16:12:24.607 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673035 2024-05-08T16:12:24.607 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673035 2024-05-08T16:12:24.607 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836565 1-42949673035' 2024-05-08T16:12:24.607 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T16:12:24.607 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-05-08T16:12:24.749 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509510 2024-05-08T16:12:24.897 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509510 2024-05-08T16:12:24.897 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836565 1-42949673035 2-64424509510' 2024-05-08T16:12:24.897 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T16:12:24.897 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-05-08T16:12:24.898 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=85899345983 2024-05-08T16:12:24.899 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 85899345983 2024-05-08T16:12:24.899 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836565 1-42949673035 2-64424509510 3-85899345983' 2024-05-08T16:12:24.899 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T16:12:24.899 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-05-08T16:12:25.055 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182463 2024-05-08T16:12:25.055 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182463 2024-05-08T16:12:25.055 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836565 1-42949673035 2-64424509510 3-85899345983 4-107374182463' 2024-05-08T16:12:25.055 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T16:12:25.055 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-05-08T16:12:25.209 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=128849018924 2024-05-08T16:12:25.209 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 128849018924 2024-05-08T16:12:25.209 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836565 1-42949673035 2-64424509510 3-85899345983 4-107374182463 5-128849018924' 2024-05-08T16:12:25.209 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T16:12:25.209 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-05-08T16:12:25.359 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855398 2024-05-08T16:12:25.359 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855398 2024-05-08T16:12:25.360 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836565 1-42949673035 2-64424509510 3-85899345983 4-107374182463 5-128849018924 6-150323855398' 2024-05-08T16:12:25.360 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T16:12:25.360 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-05-08T16:12:25.512 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691870 2024-05-08T16:12:25.512 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691870 2024-05-08T16:12:25.512 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836565 1-42949673035 2-64424509510 3-85899345983 4-107374182463 5-128849018924 6-150323855398 7-171798691870' 2024-05-08T16:12:25.512 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T16:12:25.513 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-05-08T16:12:25.663 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528345 2024-05-08T16:12:25.663 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528345 2024-05-08T16:12:25.663 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836565 1-42949673035 2-64424509510 3-85899345983 4-107374182463 5-128849018924 6-150323855398 7-171798691870 8-193273528345' 2024-05-08T16:12:25.663 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T16:12:25.663 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-05-08T16:12:25.815 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=214748364817 2024-05-08T16:12:25.815 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 214748364817 2024-05-08T16:12:25.815 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836565 1-42949673035 2-64424509510 3-85899345983 4-107374182463 5-128849018924 6-150323855398 7-171798691870 8-193273528345 9-214748364817' 2024-05-08T16:12:25.815 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T16:12:25.815 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.10 flush_pg_stats 2024-05-08T16:12:25.971 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=236223201290 2024-05-08T16:12:25.971 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 236223201290 2024-05-08T16:12:25.971 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836565 1-42949673035 2-64424509510 3-85899345983 4-107374182463 5-128849018924 6-150323855398 7-171798691870 8-193273528345 9-214748364817 10-236223201290' 2024-05-08T16:12:25.971 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-08T16:12:25.972 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836565 2024-05-08T16:12:25.972 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-08T16:12:25.972 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-05-08T16:12:25.973 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836565 2024-05-08T16:12:25.974 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-08T16:12:25.975 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836565 2024-05-08T16:12:25.975 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836565' 2024-05-08T16:12:25.975 INFO:tasks.workunit.client.0.smithi172.stdout:waiting osd.0 seq 21474836565 2024-05-08T16:12:25.976 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-05-08T16:12:26.393 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836565 -lt 21474836565 2024-05-08T16:12:26.393 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-08T16:12:26.394 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673035 2024-05-08T16:12:26.394 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-08T16:12:26.395 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-05-08T16:12:26.396 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673035 2024-05-08T16:12:26.396 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-08T16:12:26.397 INFO:tasks.workunit.client.0.smithi172.stdout:waiting osd.1 seq 42949673035 2024-05-08T16:12:26.398 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673035 2024-05-08T16:12:26.398 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673035' 2024-05-08T16:12:26.398 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-05-08T16:12:26.803 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673036 -lt 42949673035 2024-05-08T16:12:26.803 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-08T16:12:26.804 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509510 2024-05-08T16:12:26.804 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-08T16:12:26.806 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-05-08T16:12:26.807 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509510 2024-05-08T16:12:26.807 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-08T16:12:26.808 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509510 2024-05-08T16:12:26.808 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509510' 2024-05-08T16:12:26.808 INFO:tasks.workunit.client.0.smithi172.stdout:waiting osd.2 seq 64424509510 2024-05-08T16:12:26.808 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-05-08T16:12:27.333 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509510 -lt 64424509510 2024-05-08T16:12:27.333 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-08T16:12:27.334 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-85899345983 2024-05-08T16:12:27.334 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-08T16:12:27.335 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-05-08T16:12:27.336 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-85899345983 2024-05-08T16:12:27.337 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-08T16:12:27.338 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=85899345983 2024-05-08T16:12:27.338 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 85899345983' 2024-05-08T16:12:27.339 INFO:tasks.workunit.client.0.smithi172.stdout:waiting osd.3 seq 85899345983 2024-05-08T16:12:27.341 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-05-08T16:12:27.980 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 85899345984 -lt 85899345983 2024-05-08T16:12:27.980 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-08T16:12:27.981 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182463 2024-05-08T16:12:27.981 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-08T16:12:27.983 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-05-08T16:12:27.984 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182463 2024-05-08T16:12:27.984 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-08T16:12:27.985 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182463 2024-05-08T16:12:27.985 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182463' 2024-05-08T16:12:27.985 INFO:tasks.workunit.client.0.smithi172.stdout:waiting osd.4 seq 107374182463 2024-05-08T16:12:27.985 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-05-08T16:12:28.438 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182463 -lt 107374182463 2024-05-08T16:12:28.438 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-08T16:12:28.439 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-08T16:12:28.439 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-128849018924 2024-05-08T16:12:28.439 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-05-08T16:12:28.440 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-128849018924 2024-05-08T16:12:28.440 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-08T16:12:28.441 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=128849018924 2024-05-08T16:12:28.441 INFO:tasks.workunit.client.0.smithi172.stdout:waiting osd.5 seq 128849018924 2024-05-08T16:12:28.442 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 128849018924' 2024-05-08T16:12:28.442 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-05-08T16:12:28.960 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 128849018925 -lt 128849018924 2024-05-08T16:12:28.961 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-08T16:12:28.961 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855398 2024-05-08T16:12:28.961 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-08T16:12:28.962 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-05-08T16:12:28.963 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855398 2024-05-08T16:12:28.963 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-08T16:12:28.964 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855398 2024-05-08T16:12:28.964 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855398' 2024-05-08T16:12:28.965 INFO:tasks.workunit.client.0.smithi172.stdout:waiting osd.6 seq 150323855398 2024-05-08T16:12:28.965 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-05-08T16:12:29.394 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855399 -lt 150323855398 2024-05-08T16:12:29.394 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-08T16:12:29.395 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691870 2024-05-08T16:12:29.395 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-08T16:12:29.396 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-05-08T16:12:29.397 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691870 2024-05-08T16:12:29.397 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-08T16:12:29.398 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691870 2024-05-08T16:12:29.399 INFO:tasks.workunit.client.0.smithi172.stdout:waiting osd.7 seq 171798691870 2024-05-08T16:12:29.399 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691870' 2024-05-08T16:12:29.399 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-05-08T16:12:29.817 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691871 -lt 171798691870 2024-05-08T16:12:29.817 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-08T16:12:29.818 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-193273528345 2024-05-08T16:12:29.818 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-08T16:12:29.819 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-05-08T16:12:29.820 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528345 2024-05-08T16:12:29.820 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-08T16:12:29.821 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528345 2024-05-08T16:12:29.821 INFO:tasks.workunit.client.0.smithi172.stdout:waiting osd.8 seq 193273528345 2024-05-08T16:12:29.821 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528345' 2024-05-08T16:12:29.821 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-05-08T16:12:30.269 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528346 -lt 193273528345 2024-05-08T16:12:30.269 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-08T16:12:30.269 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-08T16:12:30.269 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-214748364817 2024-05-08T16:12:30.270 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-05-08T16:12:30.271 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-214748364817 2024-05-08T16:12:30.272 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-08T16:12:30.272 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=214748364817 2024-05-08T16:12:30.272 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 214748364817' 2024-05-08T16:12:30.273 INFO:tasks.workunit.client.0.smithi172.stdout:waiting osd.9 seq 214748364817 2024-05-08T16:12:30.273 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-05-08T16:12:30.706 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 214748364818 -lt 214748364817 2024-05-08T16:12:30.706 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-08T16:12:30.707 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 10-236223201290 2024-05-08T16:12:30.707 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-08T16:12:30.708 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=10 2024-05-08T16:12:30.709 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 10-236223201290 2024-05-08T16:12:30.709 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-08T16:12:30.710 INFO:tasks.workunit.client.0.smithi172.stdout:waiting osd.10 seq 236223201290 2024-05-08T16:12:30.711 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=236223201290 2024-05-08T16:12:30.711 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.10 seq 236223201290' 2024-05-08T16:12:30.711 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 10 2024-05-08T16:12:31.146 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 236223201291 -lt 236223201290 2024-05-08T16:12:31.147 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-05-08T16:12:31.148 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-05-08T16:12:31.148 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-05-08T16:12:31.696 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-05-08T16:12:31.696 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-05-08T16:12:31.697 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-05-08T16:12:31.698 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-05-08T16:12:31.698 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-05-08T16:12:31.698 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-05-08T16:12:31.698 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-05-08T16:12:31.698 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-05-08T16:12:32.130 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-05-08T16:12:32.130 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-05-08T16:12:32.130 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-05-08T16:12:32.130 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-05-08T16:12:32.673 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-05-08T16:12:32.673 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-05-08T16:12:32.673 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-05-08T16:12:32.675 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:42: run: get_asok_path osd.0 2024-05-08T16:12:32.675 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-05-08T16:12:32.675 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-05-08T16:12:32.675 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-08T16:12:32.675 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T16:12:32.675 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.48735 2024-05-08T16:12:32.675 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.48735/ceph-osd.0.asok 2024-05-08T16:12:32.676 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:42: run: CEPH_ARGS= 2024-05-08T16:12:32.676 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:42: run: ceph --admin-daemon /tmp/ceph-asok.48735/ceph-osd.0.asok log flush 2024-05-08T16:12:32.772 INFO:tasks.workunit.client.0.smithi172.stdout:{} 2024-05-08T16:12:32.784 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:43: run: grep 'load: jerasure.*lrc' td/test-erasure-code/osd.0.log 2024-05-08T16:12:32.791 INFO:tasks.workunit.client.0.smithi172.stdout:2024-05-08T16:05:25.771+0000 7f44dd1e37c0 0 load: jerasure load: lrc 2024-05-08T16:12:32.791 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:44: run: create_erasure_coded_pool ecpool 2024-05-08T16:12:32.791 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:56: create_erasure_coded_pool: local poolname=ecpool 2024-05-08T16:12:32.792 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:58: create_erasure_coded_pool: ceph osd erasure-code-profile set myprofile crush-failure-domain=osd 2024-05-08T16:12:33.466 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:60: create_erasure_coded_pool: create_pool ecpool 12 12 erasure myprofile 2024-05-08T16:12:33.466 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create ecpool 12 12 erasure myprofile 2024-05-08T16:12:34.397 INFO:tasks.workunit.client.0.smithi172.stderr:pool 'ecpool' already exists 2024-05-08T16:12:34.416 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-05-08T16:12:35.418 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:62: create_erasure_coded_pool: wait_for_clean 2024-05-08T16:12:35.418 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-05-08T16:12:35.418 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-05-08T16:12:35.418 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-05-08T16:12:35.419 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-05-08T16:12:35.420 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-05-08T16:12:35.420 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-05-08T16:12:35.420 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-05-08T16:12:35.421 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-05-08T16:12:35.421 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-05-08T16:12:35.537 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-05-08T16:12:35.538 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-05-08T16:12:35.538 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-05-08T16:12:35.538 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-05-08T16:12:35.538 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-05-08T16:12:35.538 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-05-08T16:12:36.497 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-05-08T16:12:36.498 INFO:tasks.workunit.client.0.smithi172.stderr:1 2024-05-08T16:12:36.498 INFO:tasks.workunit.client.0.smithi172.stderr:2 2024-05-08T16:12:36.498 INFO:tasks.workunit.client.0.smithi172.stderr:3 2024-05-08T16:12:36.498 INFO:tasks.workunit.client.0.smithi172.stderr:4 2024-05-08T16:12:36.498 INFO:tasks.workunit.client.0.smithi172.stderr:5 2024-05-08T16:12:36.498 INFO:tasks.workunit.client.0.smithi172.stderr:6 2024-05-08T16:12:36.498 INFO:tasks.workunit.client.0.smithi172.stderr:7 2024-05-08T16:12:36.498 INFO:tasks.workunit.client.0.smithi172.stderr:8 2024-05-08T16:12:36.498 INFO:tasks.workunit.client.0.smithi172.stderr:9 2024-05-08T16:12:36.498 INFO:tasks.workunit.client.0.smithi172.stderr:10' 2024-05-08T16:12:36.498 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-05-08T16:12:36.498 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T16:12:36.498 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-05-08T16:12:36.910 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836569 2024-05-08T16:12:36.910 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836569 2024-05-08T16:12:36.910 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836569' 2024-05-08T16:12:36.910 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T16:12:36.911 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-05-08T16:12:37.066 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673040 2024-05-08T16:12:37.066 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673040 2024-05-08T16:12:37.066 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836569 1-42949673040' 2024-05-08T16:12:37.066 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T16:12:37.066 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-05-08T16:12:37.571 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509514 2024-05-08T16:12:37.571 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509514 2024-05-08T16:12:37.571 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836569 1-42949673040 2-64424509514' 2024-05-08T16:12:37.571 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T16:12:37.571 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-05-08T16:12:37.995 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=85899345988 2024-05-08T16:12:37.995 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 85899345988 2024-05-08T16:12:37.995 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836569 1-42949673040 2-64424509514 3-85899345988' 2024-05-08T16:12:37.995 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T16:12:37.996 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-05-08T16:12:38.417 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182467 2024-05-08T16:12:38.417 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182467 2024-05-08T16:12:38.417 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836569 1-42949673040 2-64424509514 3-85899345988 4-107374182467' 2024-05-08T16:12:38.417 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T16:12:38.418 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-05-08T16:12:38.779 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=128849018929 2024-05-08T16:12:38.780 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 128849018929 2024-05-08T16:12:38.780 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836569 1-42949673040 2-64424509514 3-85899345988 4-107374182467 5-128849018929' 2024-05-08T16:12:38.780 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T16:12:38.780 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-05-08T16:12:39.002 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855403 2024-05-08T16:12:39.002 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855403 2024-05-08T16:12:39.002 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836569 1-42949673040 2-64424509514 3-85899345988 4-107374182467 5-128849018929 6-150323855403' 2024-05-08T16:12:39.002 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T16:12:39.002 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-05-08T16:12:39.165 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691875 2024-05-08T16:12:39.166 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691875 2024-05-08T16:12:39.166 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836569 1-42949673040 2-64424509514 3-85899345988 4-107374182467 5-128849018929 6-150323855403 7-171798691875' 2024-05-08T16:12:39.166 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T16:12:39.166 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-05-08T16:12:39.352 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528350 2024-05-08T16:12:39.352 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528350 2024-05-08T16:12:39.352 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836569 1-42949673040 2-64424509514 3-85899345988 4-107374182467 5-128849018929 6-150323855403 7-171798691875 8-193273528350' 2024-05-08T16:12:39.352 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T16:12:39.352 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-05-08T16:12:39.612 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=214748364821 2024-05-08T16:12:39.612 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 214748364821 2024-05-08T16:12:39.612 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836569 1-42949673040 2-64424509514 3-85899345988 4-107374182467 5-128849018929 6-150323855403 7-171798691875 8-193273528350 9-214748364821' 2024-05-08T16:12:39.612 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T16:12:39.612 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.10 flush_pg_stats 2024-05-08T16:12:39.771 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=236223201295 2024-05-08T16:12:39.772 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 236223201295 2024-05-08T16:12:39.772 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836569 1-42949673040 2-64424509514 3-85899345988 4-107374182467 5-128849018929 6-150323855403 7-171798691875 8-193273528350 9-214748364821 10-236223201295' 2024-05-08T16:12:39.772 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-08T16:12:39.773 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836569 2024-05-08T16:12:39.773 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-08T16:12:39.773 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-05-08T16:12:39.774 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836569 2024-05-08T16:12:39.774 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-08T16:12:39.775 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836569 2024-05-08T16:12:39.776 INFO:tasks.workunit.client.0.smithi172.stdout:waiting osd.0 seq 21474836569 2024-05-08T16:12:39.776 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836569' 2024-05-08T16:12:39.777 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-05-08T16:12:40.231 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836569 -lt 21474836569 2024-05-08T16:12:40.231 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-08T16:12:40.232 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-08T16:12:40.232 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673040 2024-05-08T16:12:40.233 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-05-08T16:12:40.234 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673040 2024-05-08T16:12:40.234 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-08T16:12:40.235 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673040 2024-05-08T16:12:40.235 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673040' 2024-05-08T16:12:40.235 INFO:tasks.workunit.client.0.smithi172.stdout:waiting osd.1 seq 42949673040 2024-05-08T16:12:40.236 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-05-08T16:12:40.644 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673040 -lt 42949673040 2024-05-08T16:12:40.644 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-08T16:12:40.645 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509514 2024-05-08T16:12:40.645 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-08T16:12:40.646 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-05-08T16:12:40.647 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509514 2024-05-08T16:12:40.647 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-08T16:12:40.648 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509514 2024-05-08T16:12:40.649 INFO:tasks.workunit.client.0.smithi172.stdout:waiting osd.2 seq 64424509514 2024-05-08T16:12:40.649 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509514' 2024-05-08T16:12:40.649 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-05-08T16:12:41.153 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509515 -lt 64424509514 2024-05-08T16:12:41.153 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-08T16:12:41.154 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-85899345988 2024-05-08T16:12:41.154 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-08T16:12:41.156 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-05-08T16:12:41.157 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-85899345988 2024-05-08T16:12:41.157 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-08T16:12:41.158 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=85899345988 2024-05-08T16:12:41.158 INFO:tasks.workunit.client.0.smithi172.stdout:waiting osd.3 seq 85899345988 2024-05-08T16:12:41.159 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 85899345988' 2024-05-08T16:12:41.159 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-05-08T16:12:41.617 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 85899345988 -lt 85899345988 2024-05-08T16:12:41.618 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-08T16:12:41.618 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182467 2024-05-08T16:12:41.619 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-08T16:12:41.620 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-05-08T16:12:41.621 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182467 2024-05-08T16:12:41.622 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-08T16:12:41.623 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182467 2024-05-08T16:12:41.623 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182467' 2024-05-08T16:12:41.623 INFO:tasks.workunit.client.0.smithi172.stdout:waiting osd.4 seq 107374182467 2024-05-08T16:12:41.623 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-05-08T16:12:42.042 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182468 -lt 107374182467 2024-05-08T16:12:42.043 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-08T16:12:42.044 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-128849018929 2024-05-08T16:12:42.044 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-08T16:12:42.046 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-05-08T16:12:42.047 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-128849018929 2024-05-08T16:12:42.047 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-08T16:12:42.048 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=128849018929 2024-05-08T16:12:42.048 INFO:tasks.workunit.client.0.smithi172.stdout:waiting osd.5 seq 128849018929 2024-05-08T16:12:42.048 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 128849018929' 2024-05-08T16:12:42.048 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-05-08T16:12:42.477 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 128849018929 -lt 128849018929 2024-05-08T16:12:42.477 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-08T16:12:42.478 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855403 2024-05-08T16:12:42.478 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-08T16:12:42.479 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-05-08T16:12:42.480 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855403 2024-05-08T16:12:42.480 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-08T16:12:42.482 INFO:tasks.workunit.client.0.smithi172.stdout:waiting osd.6 seq 150323855403 2024-05-08T16:12:42.482 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855403 2024-05-08T16:12:42.482 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855403' 2024-05-08T16:12:42.482 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-05-08T16:12:42.909 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855404 -lt 150323855403 2024-05-08T16:12:42.910 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-08T16:12:42.911 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691875 2024-05-08T16:12:42.911 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-08T16:12:42.912 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-05-08T16:12:42.913 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691875 2024-05-08T16:12:42.913 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-08T16:12:42.914 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691875 2024-05-08T16:12:42.915 INFO:tasks.workunit.client.0.smithi172.stdout:waiting osd.7 seq 171798691875 2024-05-08T16:12:42.915 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691875' 2024-05-08T16:12:42.915 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-05-08T16:12:43.363 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691876 -lt 171798691875 2024-05-08T16:12:43.363 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-08T16:12:43.364 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-193273528350 2024-05-08T16:12:43.364 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-08T16:12:43.365 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-05-08T16:12:43.366 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528350 2024-05-08T16:12:43.366 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-08T16:12:43.368 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528350 2024-05-08T16:12:43.368 INFO:tasks.workunit.client.0.smithi172.stdout:waiting osd.8 seq 193273528350 2024-05-08T16:12:43.368 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528350' 2024-05-08T16:12:43.368 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-05-08T16:12:43.816 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528351 -lt 193273528350 2024-05-08T16:12:43.816 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-08T16:12:43.818 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-214748364821 2024-05-08T16:12:43.818 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-08T16:12:43.820 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-05-08T16:12:43.820 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-214748364821 2024-05-08T16:12:43.821 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-08T16:12:43.822 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=214748364821 2024-05-08T16:12:43.822 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 214748364821' 2024-05-08T16:12:43.822 INFO:tasks.workunit.client.0.smithi172.stdout:waiting osd.9 seq 214748364821 2024-05-08T16:12:43.823 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-05-08T16:12:44.244 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 214748364822 -lt 214748364821 2024-05-08T16:12:44.244 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-08T16:12:44.245 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-08T16:12:44.245 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 10-236223201295 2024-05-08T16:12:44.246 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=10 2024-05-08T16:12:44.247 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 10-236223201295 2024-05-08T16:12:44.247 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-08T16:12:44.248 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=236223201295 2024-05-08T16:12:44.249 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.10 seq 236223201295' 2024-05-08T16:12:44.249 INFO:tasks.workunit.client.0.smithi172.stdout:waiting osd.10 seq 236223201295 2024-05-08T16:12:44.249 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 10 2024-05-08T16:12:44.663 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 236223201296 -lt 236223201295 2024-05-08T16:12:44.663 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-05-08T16:12:44.664 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-05-08T16:12:44.664 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-05-08T16:12:45.235 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 16 == 0 2024-05-08T16:12:45.235 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-05-08T16:12:45.236 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-05-08T16:12:45.236 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-05-08T16:12:45.236 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-05-08T16:12:45.236 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-05-08T16:12:45.236 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-05-08T16:12:45.236 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-05-08T16:12:45.693 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=16 2024-05-08T16:12:45.694 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-05-08T16:12:45.694 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-05-08T16:12:45.694 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-05-08T16:12:46.240 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 16 = 16 2024-05-08T16:12:46.240 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-05-08T16:12:46.240 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-05-08T16:12:46.240 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:46: run: set 2024-05-08T16:12:46.240 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:46: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-05-08T16:12:46.243 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:46: run: local 'funcs=TEST_alignment_constraints 2024-05-08T16:12:46.243 INFO:tasks.workunit.client.0.smithi172.stderr:TEST_chunk_mapping 2024-05-08T16:12:46.243 INFO:tasks.workunit.client.0.smithi172.stderr:TEST_rados_put_get_isa 2024-05-08T16:12:46.243 INFO:tasks.workunit.client.0.smithi172.stderr:TEST_rados_put_get_jerasure 2024-05-08T16:12:46.243 INFO:tasks.workunit.client.0.smithi172.stderr:TEST_rados_put_get_lrc_advanced 2024-05-08T16:12:46.243 INFO:tasks.workunit.client.0.smithi172.stderr:TEST_rados_put_get_lrc_kml 2024-05-08T16:12:46.243 INFO:tasks.workunit.client.0.smithi172.stderr:TEST_rados_put_get_shec' 2024-05-08T16:12:46.243 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:47: run: for func in $funcs 2024-05-08T16:12:46.244 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:48: run: TEST_alignment_constraints td/test-erasure-code 2024-05-08T16:12:46.244 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:253: TEST_alignment_constraints: local payload=ABC 2024-05-08T16:12:46.244 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:254: TEST_alignment_constraints: echo ABC 2024-05-08T16:12:46.244 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:260: TEST_alignment_constraints: ceph-conf --show-config-value osd_pool_erasure_code_stripe_unit 2024-05-08T16:12:46.264 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:260: TEST_alignment_constraints: local stripe_unit=4096 2024-05-08T16:12:46.265 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:261: TEST_alignment_constraints: ceph osd erasure-code-profile get myprofile 2024-05-08T16:12:46.265 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:261: TEST_alignment_constraints: grep k= 2024-05-08T16:12:46.725 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:261: TEST_alignment_constraints: eval local k=2 2024-05-08T16:12:46.725 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:261: TEST_alignment_constraints: local k=2 2024-05-08T16:12:46.725 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:262: TEST_alignment_constraints: local block_size=8191 2024-05-08T16:12:46.725 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:263: TEST_alignment_constraints: dd if=/dev/zero of=td/test-erasure-code/ORIGINAL bs=8191 count=2 2024-05-08T16:12:46.725 INFO:tasks.workunit.client.0.smithi172.stderr:2+0 records in 2024-05-08T16:12:46.725 INFO:tasks.workunit.client.0.smithi172.stderr:2+0 records out 2024-05-08T16:12:46.725 INFO:tasks.workunit.client.0.smithi172.stderr:16382 bytes (16 kB, 16 KiB) copied, 0.000174032 s, 94.1 MB/s 2024-05-08T16:12:46.726 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:264: TEST_alignment_constraints: rados --block-size=8191 --pool ecpool put UNALIGNED td/test-erasure-code/ORIGINAL 2024-05-08T16:12:46.752 INFO:tasks.workunit.client.0.smithi172.stderr:INFO: op_size has been rounded to 8192 2024-05-08T16:12:47.245 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:266: TEST_alignment_constraints: rm td/test-erasure-code/ORIGINAL 2024-05-08T16:12:47.246 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:47: run: for func in $funcs 2024-05-08T16:12:47.247 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:48: run: TEST_chunk_mapping td/test-erasure-code 2024-05-08T16:12:47.247 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:304: TEST_chunk_mapping: local dir=td/test-erasure-code 2024-05-08T16:12:47.247 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:311: TEST_chunk_mapping: verify_chunk_mapping td/test-erasure-code ecpool 0 1 2024-05-08T16:12:47.247 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:281: verify_chunk_mapping: local dir=td/test-erasure-code 2024-05-08T16:12:47.247 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:282: verify_chunk_mapping: local poolname=ecpool 2024-05-08T16:12:47.247 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:283: verify_chunk_mapping: local first=0 2024-05-08T16:12:47.247 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:284: verify_chunk_mapping: local second=1 2024-05-08T16:12:47.247 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:286: verify_chunk_mapping: chunk_size 2024-05-08T16:12:47.248 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:270: chunk_size: ceph-conf --show-config-value osd_pool_erasure_code_stripe_unit 2024-05-08T16:12:47.269 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:270: chunk_size: echo 4096 2024-05-08T16:12:47.269 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:286: verify_chunk_mapping: printf '%*s' 4096 FIRSTecpool 2024-05-08T16:12:47.270 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:286: verify_chunk_mapping: chunk_size 2024-05-08T16:12:47.270 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:270: chunk_size: ceph-conf --show-config-value osd_pool_erasure_code_stripe_unit 2024-05-08T16:12:47.292 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:270: chunk_size: echo 4096 2024-05-08T16:12:47.293 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:286: verify_chunk_mapping: printf '%*s' 4096 SECONDecpool 2024-05-08T16:12:47.293 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:286: verify_chunk_mapping: local 'payload= FIRSTecpool SECONDecpool' 2024-05-08T16:12:47.294 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:287: verify_chunk_mapping: echo -n ' FIRSTecpool SECONDecpool' 2024-05-08T16:12:47.294 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:289: verify_chunk_mapping: rados --pool ecpool put SOMETHINGecpool td/test-erasure-code/ORIGINAL 2024-05-08T16:12:47.556 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:290: verify_chunk_mapping: rados --pool ecpool get SOMETHINGecpool td/test-erasure-code/COPY 2024-05-08T16:12:47.696 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:291: verify_chunk_mapping: get_osds ecpool SOMETHINGecpool 2024-05-08T16:12:47.696 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=ecpool 2024-05-08T16:12:47.696 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=SOMETHINGecpool 2024-05-08T16:12:47.697 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map ecpool SOMETHINGecpool 2024-05-08T16:12:47.697 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-05-08T16:12:48.121 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-05-08T16:12:48.121 INFO:tasks.workunit.client.0.smithi172.stderr:5 2024-05-08T16:12:48.122 INFO:tasks.workunit.client.0.smithi172.stderr:9 2024-05-08T16:12:48.122 INFO:tasks.workunit.client.0.smithi172.stderr:0' 2024-05-08T16:12:48.122 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 5 9 0 2024-05-08T16:12:48.122 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:291: verify_chunk_mapping: osds=('3' '5' '9' '0') 2024-05-08T16:12:48.122 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:291: verify_chunk_mapping: local -a osds 2024-05-08T16:12:48.122 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:292: verify_chunk_mapping: (( i = 0 )) 2024-05-08T16:12:48.122 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:292: verify_chunk_mapping: (( i < 4 )) 2024-05-08T16:12:48.122 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:293: verify_chunk_mapping: ceph daemon osd.3 flush_journal 2024-05-08T16:12:48.252 INFO:tasks.workunit.client.0.smithi172.stderr:Can't get admin socket path: unable to get conf option admin_socket for osd: b"error parsing 'osd': expected string of the form TYPE.ID, valid types are: auth, mon, osd, mds, mgr, client\n" 2024-05-08T16:12:48.253 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:292: verify_chunk_mapping: (( i++ )) 2024-05-08T16:12:48.253 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:292: verify_chunk_mapping: (( i < 4 )) 2024-05-08T16:12:48.253 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:293: verify_chunk_mapping: ceph daemon osd.5 flush_journal 2024-05-08T16:12:48.383 INFO:tasks.workunit.client.0.smithi172.stderr:Can't get admin socket path: unable to get conf option admin_socket for osd: b"error parsing 'osd': expected string of the form TYPE.ID, valid types are: auth, mon, osd, mds, mgr, client\n" 2024-05-08T16:12:48.385 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:292: verify_chunk_mapping: (( i++ )) 2024-05-08T16:12:48.385 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:292: verify_chunk_mapping: (( i < 4 )) 2024-05-08T16:12:48.385 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:293: verify_chunk_mapping: ceph daemon osd.9 flush_journal 2024-05-08T16:12:48.516 INFO:tasks.workunit.client.0.smithi172.stderr:Can't get admin socket path: unable to get conf option admin_socket for osd: b"error parsing 'osd': expected string of the form TYPE.ID, valid types are: auth, mon, osd, mds, mgr, client\n" 2024-05-08T16:12:48.517 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:292: verify_chunk_mapping: (( i++ )) 2024-05-08T16:12:48.517 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:292: verify_chunk_mapping: (( i < 4 )) 2024-05-08T16:12:48.517 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:293: verify_chunk_mapping: ceph daemon osd.0 flush_journal 2024-05-08T16:12:48.648 INFO:tasks.workunit.client.0.smithi172.stderr:Can't get admin socket path: unable to get conf option admin_socket for osd: b"error parsing 'osd': expected string of the form TYPE.ID, valid types are: auth, mon, osd, mds, mgr, client\n" 2024-05-08T16:12:48.650 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:292: verify_chunk_mapping: (( i++ )) 2024-05-08T16:12:48.650 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:292: verify_chunk_mapping: (( i < 4 )) 2024-05-08T16:12:48.650 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:295: verify_chunk_mapping: diff td/test-erasure-code/ORIGINAL td/test-erasure-code/COPY 2024-05-08T16:12:48.662 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:296: verify_chunk_mapping: rm td/test-erasure-code/COPY 2024-05-08T16:12:48.664 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:298: verify_chunk_mapping: get_osds ecpool SOMETHINGecpool 2024-05-08T16:12:48.664 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=ecpool 2024-05-08T16:12:48.665 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=SOMETHINGecpool 2024-05-08T16:12:48.665 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map ecpool SOMETHINGecpool 2024-05-08T16:12:48.665 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-05-08T16:12:49.088 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-05-08T16:12:49.089 INFO:tasks.workunit.client.0.smithi172.stderr:5 2024-05-08T16:12:49.089 INFO:tasks.workunit.client.0.smithi172.stderr:9 2024-05-08T16:12:49.089 INFO:tasks.workunit.client.0.smithi172.stderr:0' 2024-05-08T16:12:49.089 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 5 9 0 2024-05-08T16:12:49.089 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:298: verify_chunk_mapping: osds=('3' '5' '9' '0') 2024-05-08T16:12:49.089 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:298: verify_chunk_mapping: local -a osds 2024-05-08T16:12:49.089 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:299: verify_chunk_mapping: objectstore_tool td/test-erasure-code 3 SOMETHINGecpool get-bytes 2024-05-08T16:12:49.089 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/test-erasure-code 2024-05-08T16:12:49.089 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:299: verify_chunk_mapping: grep --quiet FIRSTecpool 2024-05-08T16:12:49.089 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-05-08T16:12:49.089 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=3 2024-05-08T16:12:49.089 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-05-08T16:12:49.089 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/test-erasure-code 3 SOMETHINGecpool get-bytes 2024-05-08T16:12:49.090 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/test-erasure-code 2024-05-08T16:12:49.090 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-05-08T16:12:49.090 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=3 2024-05-08T16:12:49.090 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-05-08T16:12:49.090 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/test-erasure-code TERM osd.3 2024-05-08T16:12:49.090 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-08T16:12:49.090 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-08T16:12:49.090 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-08T16:12:49.090 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-08T16:12:49.090 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-08T16:12:49.397 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-08T16:12:49.397 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/test-erasure-code 3 SOMETHINGecpool get-bytes 2024-05-08T16:12:49.397 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/test-erasure-code 2024-05-08T16:12:49.397 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-05-08T16:12:49.397 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=3 2024-05-08T16:12:49.397 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-05-08T16:12:49.397 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/test-erasure-code/3 2024-05-08T16:12:49.397 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/test-erasure-code/3 SOMETHINGecpool get-bytes 2024-05-08T16:12:51.788 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/test-erasure-code 3 2024-05-08T16:12:51.788 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/test-erasure-code 2024-05-08T16:12:51.788 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-05-08T16:12:51.788 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2024-05-08T16:12:51.788 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-05-08T16:12:51.788 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/test-erasure-code/3 2024-05-08T16:12:51.788 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=f37874d1-6990-4767-8498-03a75cc31118 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false' 2024-05-08T16:12:51.788 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-08T16:12:51.788 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-05-08T16:12:51.788 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-08T16:12:51.788 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/test-erasure-code/3' 2024-05-08T16:12:51.789 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-code/3/journal' 2024-05-08T16:12:51.789 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-05-08T16:12:51.789 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-05-08T16:12:51.789 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/test-erasure-code' 2024-05-08T16:12:51.789 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-05-08T16:12:51.789 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-08T16:12:51.789 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-08T16:12:51.789 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-08T16:12:51.789 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T16:12:51.789 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.48735 2024-05-08T16:12:51.789 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.48735/$cluster-$name.asok' 2024-05-08T16:12:51.789 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.48735/$cluster-$name.asok' 2024-05-08T16:12:51.789 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-05-08T16:12:51.790 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/test-erasure-code/$name.log' 2024-05-08T16:12:51.790 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/test-erasure-code/$name.pid' 2024-05-08T16:12:51.790 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-08T16:12:51.790 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-08T16:12:51.790 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-08T16:12:51.790 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-08T16:12:51.790 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-05-08T16:12:51.790 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-05-08T16:12:51.790 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/test-erasure-code/3 2024-05-08T16:12:51.791 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2024-05-08T16:12:51.791 INFO:tasks.workunit.client.0.smithi172.stderr:start osd.3 2024-05-08T16:12:51.791 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=f37874d1-6990-4767-8498-03a75cc31118 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/3 --osd-journal=td/test-erasure-code/3/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.48735/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-05-08T16:12:51.791 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/test-erasure-code/3/whoami 2024-05-08T16:12:51.792 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2024-05-08T16:12:51.792 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-05-08T16:12:51.792 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-05-08T16:12:51.792 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-05-08T16:12:51.823 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:12:51.828+0000 7fa3cdf0b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:12:51.826 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:12:51.832+0000 7fa3cdf0b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:12:51.827 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:12:51.836+0000 7fa3cdf0b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:12:52.254 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2024-05-08T16:12:52.255 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-08T16:12:52.255 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-05-08T16:12:52.255 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-08T16:12:52.255 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-08T16:12:52.255 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:12:52.255 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-08T16:12:52.255 INFO:tasks.workunit.client.0.smithi172.stderr:0 2024-05-08T16:12:52.255 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:12:52.255 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T16:12:52.385 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:12:52.392+0000 7fa3cdf0b7c0 -1 Falling back to public interface 2024-05-08T16:12:52.661 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:12:53.663 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:12:53.663 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:12:53.663 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-08T16:12:53.663 INFO:tasks.workunit.client.0.smithi172.stderr:1 2024-05-08T16:12:53.663 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:12:53.663 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T16:12:54.094 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:12:55.096 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:12:55.096 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:12:55.096 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-08T16:12:55.096 INFO:tasks.workunit.client.0.smithi172.stderr:2 2024-05-08T16:12:55.096 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:12:55.097 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T16:12:55.208 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:12:55.216+0000 7fa3cdf0b7c0 -1 osd.3 72 log_to_monitors true 2024-05-08T16:12:55.579 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:12:56.582 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:12:56.583 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:12:56.583 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-05-08T16:12:56.583 INFO:tasks.workunit.client.0.smithi172.stderr:3 2024-05-08T16:12:56.583 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:12:56.583 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T16:12:57.413 INFO:tasks.workunit.client.0.smithi172.stderr:osd.3 up in weight 1 up_from 76 up_thru 76 down_at 73 last_clean_interval [20,72) [v2:127.0.0.1:6826/3799404924,v1:127.0.0.1:6827/3799404924] [v2:127.0.0.1:6828/3799404924,v1:127.0.0.1:6829/3799404924] exists,up a0610a6e-a18c-4a81-9882-d0ee54a891b7 2024-05-08T16:12:57.413 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-08T16:12:57.413 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-08T16:12:57.413 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-08T16:12:57.414 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-05-08T16:12:57.414 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-05-08T16:12:57.414 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-05-08T16:12:57.414 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-05-08T16:12:57.415 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-05-08T16:12:57.415 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-05-08T16:12:57.415 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-05-08T16:12:57.415 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-05-08T16:12:57.415 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-05-08T16:12:57.415 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-05-08T16:12:57.519 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-05-08T16:12:57.519 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-05-08T16:12:57.519 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-05-08T16:12:57.520 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-05-08T16:12:57.520 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-05-08T16:12:57.520 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-05-08T16:12:57.984 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-05-08T16:12:57.984 INFO:tasks.workunit.client.0.smithi172.stderr:1 2024-05-08T16:12:57.984 INFO:tasks.workunit.client.0.smithi172.stderr:2 2024-05-08T16:12:57.984 INFO:tasks.workunit.client.0.smithi172.stderr:3 2024-05-08T16:12:57.984 INFO:tasks.workunit.client.0.smithi172.stderr:4 2024-05-08T16:12:57.984 INFO:tasks.workunit.client.0.smithi172.stderr:5 2024-05-08T16:12:57.984 INFO:tasks.workunit.client.0.smithi172.stderr:6 2024-05-08T16:12:57.984 INFO:tasks.workunit.client.0.smithi172.stderr:7 2024-05-08T16:12:57.984 INFO:tasks.workunit.client.0.smithi172.stderr:8 2024-05-08T16:12:57.984 INFO:tasks.workunit.client.0.smithi172.stderr:9 2024-05-08T16:12:57.985 INFO:tasks.workunit.client.0.smithi172.stderr:10' 2024-05-08T16:12:57.985 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-05-08T16:12:57.985 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T16:12:57.985 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-05-08T16:12:58.191 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836575 2024-05-08T16:12:58.191 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836575 2024-05-08T16:12:58.191 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836575' 2024-05-08T16:12:58.191 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T16:12:58.191 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-05-08T16:12:58.347 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673046 2024-05-08T16:12:58.347 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673046 2024-05-08T16:12:58.347 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836575 1-42949673046' 2024-05-08T16:12:58.347 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T16:12:58.347 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-05-08T16:12:58.502 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509520 2024-05-08T16:12:58.502 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509520 2024-05-08T16:12:58.502 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836575 1-42949673046 2-64424509520' 2024-05-08T16:12:58.502 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T16:12:58.502 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-05-08T16:12:58.652 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=326417514499 2024-05-08T16:12:58.652 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 326417514499 2024-05-08T16:12:58.652 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836575 1-42949673046 2-64424509520 3-326417514499' 2024-05-08T16:12:58.652 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T16:12:58.653 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-05-08T16:12:58.811 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182474 2024-05-08T16:12:58.812 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182474 2024-05-08T16:12:58.812 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836575 1-42949673046 2-64424509520 3-326417514499 4-107374182474' 2024-05-08T16:12:58.812 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T16:12:58.812 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-05-08T16:12:58.964 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=128849018935 2024-05-08T16:12:58.964 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 128849018935 2024-05-08T16:12:58.964 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836575 1-42949673046 2-64424509520 3-326417514499 4-107374182474 5-128849018935' 2024-05-08T16:12:58.965 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T16:12:58.965 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-05-08T16:12:59.124 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855409 2024-05-08T16:12:59.124 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855409 2024-05-08T16:12:59.124 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836575 1-42949673046 2-64424509520 3-326417514499 4-107374182474 5-128849018935 6-150323855409' 2024-05-08T16:12:59.124 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T16:12:59.124 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-05-08T16:12:59.276 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691881 2024-05-08T16:12:59.276 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691881 2024-05-08T16:12:59.276 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836575 1-42949673046 2-64424509520 3-326417514499 4-107374182474 5-128849018935 6-150323855409 7-171798691881' 2024-05-08T16:12:59.276 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T16:12:59.277 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-05-08T16:12:59.426 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528356 2024-05-08T16:12:59.427 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528356 2024-05-08T16:12:59.427 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836575 1-42949673046 2-64424509520 3-326417514499 4-107374182474 5-128849018935 6-150323855409 7-171798691881 8-193273528356' 2024-05-08T16:12:59.427 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T16:12:59.427 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-05-08T16:12:59.578 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=214748364827 2024-05-08T16:12:59.578 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 214748364827 2024-05-08T16:12:59.578 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836575 1-42949673046 2-64424509520 3-326417514499 4-107374182474 5-128849018935 6-150323855409 7-171798691881 8-193273528356 9-214748364827' 2024-05-08T16:12:59.578 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T16:12:59.578 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.10 flush_pg_stats 2024-05-08T16:12:59.718 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=236223201301 2024-05-08T16:12:59.718 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 236223201301 2024-05-08T16:12:59.718 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836575 1-42949673046 2-64424509520 3-326417514499 4-107374182474 5-128849018935 6-150323855409 7-171798691881 8-193273528356 9-214748364827 10-236223201301' 2024-05-08T16:12:59.718 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-08T16:12:59.719 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836575 2024-05-08T16:12:59.719 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-08T16:12:59.720 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-05-08T16:12:59.720 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836575 2024-05-08T16:12:59.721 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-08T16:12:59.721 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836575 2024-05-08T16:12:59.722 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836575' 2024-05-08T16:12:59.722 INFO:tasks.workunit.client.0.smithi172.stderr:waiting osd.0 seq 21474836575 2024-05-08T16:12:59.722 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-05-08T16:13:00.143 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836575 -lt 21474836575 2024-05-08T16:13:00.143 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-08T16:13:00.143 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673046 2024-05-08T16:13:00.143 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-08T16:13:00.143 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-05-08T16:13:00.143 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673046 2024-05-08T16:13:00.143 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-08T16:13:00.144 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673046 2024-05-08T16:13:00.144 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673046' 2024-05-08T16:13:00.144 INFO:tasks.workunit.client.0.smithi172.stderr:waiting osd.1 seq 42949673046 2024-05-08T16:13:00.144 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-05-08T16:13:00.578 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673046 -lt 42949673046 2024-05-08T16:13:00.578 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-08T16:13:00.578 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509520 2024-05-08T16:13:00.579 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-08T16:13:00.581 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-05-08T16:13:00.581 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509520 2024-05-08T16:13:00.581 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-08T16:13:00.582 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509520 2024-05-08T16:13:00.582 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509520' 2024-05-08T16:13:00.582 INFO:tasks.workunit.client.0.smithi172.stderr:waiting osd.2 seq 64424509520 2024-05-08T16:13:00.583 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-05-08T16:13:00.968 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509521 -lt 64424509520 2024-05-08T16:13:00.968 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-08T16:13:00.969 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-326417514499 2024-05-08T16:13:00.969 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-08T16:13:00.970 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-05-08T16:13:00.971 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-326417514499 2024-05-08T16:13:00.971 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-08T16:13:00.972 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=326417514499 2024-05-08T16:13:00.972 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 326417514499' 2024-05-08T16:13:00.973 INFO:tasks.workunit.client.0.smithi172.stderr:waiting osd.3 seq 326417514499 2024-05-08T16:13:00.973 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-05-08T16:13:01.382 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 326417514499 -lt 326417514499 2024-05-08T16:13:01.382 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-08T16:13:01.383 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182474 2024-05-08T16:13:01.383 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-08T16:13:01.383 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-05-08T16:13:01.384 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182474 2024-05-08T16:13:01.384 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-08T16:13:01.386 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182474 2024-05-08T16:13:01.386 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182474' 2024-05-08T16:13:01.386 INFO:tasks.workunit.client.0.smithi172.stderr:waiting osd.4 seq 107374182474 2024-05-08T16:13:01.387 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-05-08T16:13:01.773 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182474 -lt 107374182474 2024-05-08T16:13:01.773 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-08T16:13:01.774 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-128849018935 2024-05-08T16:13:01.774 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-08T16:13:01.775 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-05-08T16:13:01.776 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-128849018935 2024-05-08T16:13:01.776 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-08T16:13:01.777 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=128849018935 2024-05-08T16:13:01.777 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 128849018935' 2024-05-08T16:13:01.777 INFO:tasks.workunit.client.0.smithi172.stderr:waiting osd.5 seq 128849018935 2024-05-08T16:13:01.777 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-05-08T16:13:02.219 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 128849018935 -lt 128849018935 2024-05-08T16:13:02.219 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-08T16:13:02.219 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855409 2024-05-08T16:13:02.220 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-08T16:13:02.222 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-05-08T16:13:02.222 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855409 2024-05-08T16:13:02.222 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-08T16:13:02.224 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855409 2024-05-08T16:13:02.224 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855409' 2024-05-08T16:13:02.224 INFO:tasks.workunit.client.0.smithi172.stderr:waiting osd.6 seq 150323855409 2024-05-08T16:13:02.224 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-05-08T16:13:02.647 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855410 -lt 150323855409 2024-05-08T16:13:02.647 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-08T16:13:02.648 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-08T16:13:02.648 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691881 2024-05-08T16:13:02.649 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-05-08T16:13:02.649 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691881 2024-05-08T16:13:02.649 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-08T16:13:02.650 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691881 2024-05-08T16:13:02.651 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691881' 2024-05-08T16:13:02.651 INFO:tasks.workunit.client.0.smithi172.stderr:waiting osd.7 seq 171798691881 2024-05-08T16:13:02.651 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-05-08T16:13:03.109 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691882 -lt 171798691881 2024-05-08T16:13:03.109 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-08T16:13:03.109 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-08T16:13:03.109 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-193273528356 2024-05-08T16:13:03.110 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-05-08T16:13:03.111 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528356 2024-05-08T16:13:03.111 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-08T16:13:03.112 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528356 2024-05-08T16:13:03.112 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528356' 2024-05-08T16:13:03.112 INFO:tasks.workunit.client.0.smithi172.stderr:waiting osd.8 seq 193273528356 2024-05-08T16:13:03.112 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-05-08T16:13:03.535 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528357 -lt 193273528356 2024-05-08T16:13:03.535 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-08T16:13:03.535 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-214748364827 2024-05-08T16:13:03.535 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-08T16:13:03.536 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-05-08T16:13:03.537 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-214748364827 2024-05-08T16:13:03.537 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-08T16:13:03.538 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=214748364827 2024-05-08T16:13:03.538 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 214748364827' 2024-05-08T16:13:03.538 INFO:tasks.workunit.client.0.smithi172.stderr:waiting osd.9 seq 214748364827 2024-05-08T16:13:03.539 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-05-08T16:13:03.963 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 214748364828 -lt 214748364827 2024-05-08T16:13:03.963 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-08T16:13:03.964 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 10-236223201301 2024-05-08T16:13:03.964 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-08T16:13:03.965 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=10 2024-05-08T16:13:03.966 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 10-236223201301 2024-05-08T16:13:03.966 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-08T16:13:03.967 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=236223201301 2024-05-08T16:13:03.967 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.10 seq 236223201301' 2024-05-08T16:13:03.967 INFO:tasks.workunit.client.0.smithi172.stderr:waiting osd.10 seq 236223201301 2024-05-08T16:13:03.967 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 10 2024-05-08T16:13:04.398 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 236223201301 -lt 236223201301 2024-05-08T16:13:04.398 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-05-08T16:13:04.399 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-05-08T16:13:04.399 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-05-08T16:13:04.980 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 16 == 0 2024-05-08T16:13:04.980 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-05-08T16:13:04.981 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-05-08T16:13:04.981 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-05-08T16:13:04.981 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-05-08T16:13:04.981 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-05-08T16:13:04.981 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-05-08T16:13:04.981 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-05-08T16:13:05.421 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=16 2024-05-08T16:13:05.422 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-05-08T16:13:05.422 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-05-08T16:13:05.422 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-05-08T16:13:05.961 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 16 = 16 2024-05-08T16:13:05.961 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-05-08T16:13:05.961 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-05-08T16:13:05.963 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:300: verify_chunk_mapping: objectstore_tool td/test-erasure-code 5 SOMETHINGecpool get-bytes 2024-05-08T16:13:05.963 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/test-erasure-code 2024-05-08T16:13:05.963 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-05-08T16:13:05.964 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=5 2024-05-08T16:13:05.964 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-05-08T16:13:05.964 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:300: verify_chunk_mapping: grep --quiet SECONDecpool 2024-05-08T16:13:05.964 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/test-erasure-code 5 SOMETHINGecpool get-bytes 2024-05-08T16:13:05.964 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/test-erasure-code 2024-05-08T16:13:05.964 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-05-08T16:13:05.964 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=5 2024-05-08T16:13:05.964 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-05-08T16:13:05.964 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/test-erasure-code TERM osd.5 2024-05-08T16:13:05.964 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-08T16:13:05.964 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-08T16:13:05.965 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-08T16:13:05.965 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-08T16:13:05.965 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-08T16:13:06.271 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-08T16:13:06.271 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/test-erasure-code 5 SOMETHINGecpool get-bytes 2024-05-08T16:13:06.271 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/test-erasure-code 2024-05-08T16:13:06.271 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-05-08T16:13:06.271 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=5 2024-05-08T16:13:06.271 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-05-08T16:13:06.271 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/test-erasure-code/5 2024-05-08T16:13:06.271 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/test-erasure-code/5 SOMETHINGecpool get-bytes 2024-05-08T16:13:09.551 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/test-erasure-code 5 2024-05-08T16:13:09.551 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/test-erasure-code 2024-05-08T16:13:09.551 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-05-08T16:13:09.551 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=5 2024-05-08T16:13:09.551 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-05-08T16:13:09.551 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/test-erasure-code/5 2024-05-08T16:13:09.551 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=f37874d1-6990-4767-8498-03a75cc31118 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false' 2024-05-08T16:13:09.551 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-08T16:13:09.551 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-05-08T16:13:09.552 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-08T16:13:09.552 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/test-erasure-code/5' 2024-05-08T16:13:09.552 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-code/5/journal' 2024-05-08T16:13:09.552 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-05-08T16:13:09.552 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-05-08T16:13:09.552 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/test-erasure-code' 2024-05-08T16:13:09.552 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-05-08T16:13:09.552 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-08T16:13:09.552 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-08T16:13:09.552 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-08T16:13:09.552 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T16:13:09.552 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.48735 2024-05-08T16:13:09.554 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.48735/$cluster-$name.asok' 2024-05-08T16:13:09.554 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.48735/$cluster-$name.asok' 2024-05-08T16:13:09.554 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-05-08T16:13:09.554 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/test-erasure-code/$name.log' 2024-05-08T16:13:09.554 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/test-erasure-code/$name.pid' 2024-05-08T16:13:09.554 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-08T16:13:09.554 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-08T16:13:09.554 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-08T16:13:09.554 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-08T16:13:09.555 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-05-08T16:13:09.555 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-05-08T16:13:09.555 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/test-erasure-code/5 2024-05-08T16:13:09.555 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.5 2024-05-08T16:13:09.555 INFO:tasks.workunit.client.0.smithi172.stderr:start osd.5 2024-05-08T16:13:09.555 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 5 --fsid=f37874d1-6990-4767-8498-03a75cc31118 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/5 --osd-journal=td/test-erasure-code/5/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.48735/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-05-08T16:13:09.555 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/test-erasure-code/5/whoami 2024-05-08T16:13:09.556 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 5 = 5 ']' 2024-05-08T16:13:09.556 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-05-08T16:13:09.556 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-05-08T16:13:09.556 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-05-08T16:13:09.587 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:13:09.592+0000 7fcfdb60b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:13:09.590 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:13:09.596+0000 7fcfdb60b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:13:09.591 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:13:09.596+0000 7fcfdb60b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:13:09.991 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 5 2024-05-08T16:13:09.991 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-08T16:13:09.991 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=5 2024-05-08T16:13:09.991 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-08T16:13:09.991 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-08T16:13:09.991 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:13:09.991 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-08T16:13:09.991 INFO:tasks.workunit.client.0.smithi172.stderr:0 2024-05-08T16:13:09.992 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:13:09.992 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-05-08T16:13:10.173 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:13:10.180+0000 7fcfdb60b7c0 -1 Falling back to public interface 2024-05-08T16:13:10.420 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:13:11.423 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:13:11.423 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:13:11.423 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-08T16:13:11.423 INFO:tasks.workunit.client.0.smithi172.stderr:1 2024-05-08T16:13:11.423 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:13:11.423 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-05-08T16:13:11.823 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:13:12.824 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:13:12.825 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:13:12.825 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-08T16:13:12.825 INFO:tasks.workunit.client.0.smithi172.stderr:2 2024-05-08T16:13:12.825 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:13:12.825 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-05-08T16:13:12.834 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:13:12.840+0000 7fcfdb60b7c0 -1 osd.5 78 log_to_monitors true 2024-05-08T16:13:13.272 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:13:14.274 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:13:14.274 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:13:14.274 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-05-08T16:13:14.274 INFO:tasks.workunit.client.0.smithi172.stderr:3 2024-05-08T16:13:14.274 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:13:14.275 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-05-08T16:13:15.019 INFO:tasks.workunit.client.0.smithi172.stderr:osd.5 up in weight 1 up_from 83 up_thru 83 down_at 79 last_clean_interval [30,78) [v2:127.0.0.1:6842/2463341200,v1:127.0.0.1:6843/2463341200] [v2:127.0.0.1:6844/2463341200,v1:127.0.0.1:6845/2463341200] exists,up 608e9ac9-5b3f-4c5a-aa1f-92dc47044545 2024-05-08T16:13:15.019 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-08T16:13:15.019 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-08T16:13:15.019 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-08T16:13:15.020 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-05-08T16:13:15.020 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-05-08T16:13:15.020 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-05-08T16:13:15.020 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-05-08T16:13:15.020 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-05-08T16:13:15.021 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-05-08T16:13:15.021 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-05-08T16:13:15.021 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-05-08T16:13:15.021 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-05-08T16:13:15.021 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-05-08T16:13:15.127 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-05-08T16:13:15.127 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-05-08T16:13:15.127 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-05-08T16:13:15.127 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-05-08T16:13:15.128 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-05-08T16:13:15.128 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-05-08T16:13:15.990 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-05-08T16:13:15.990 INFO:tasks.workunit.client.0.smithi172.stderr:1 2024-05-08T16:13:15.990 INFO:tasks.workunit.client.0.smithi172.stderr:2 2024-05-08T16:13:15.990 INFO:tasks.workunit.client.0.smithi172.stderr:3 2024-05-08T16:13:15.990 INFO:tasks.workunit.client.0.smithi172.stderr:4 2024-05-08T16:13:15.990 INFO:tasks.workunit.client.0.smithi172.stderr:5 2024-05-08T16:13:15.990 INFO:tasks.workunit.client.0.smithi172.stderr:6 2024-05-08T16:13:15.990 INFO:tasks.workunit.client.0.smithi172.stderr:7 2024-05-08T16:13:15.990 INFO:tasks.workunit.client.0.smithi172.stderr:8 2024-05-08T16:13:15.990 INFO:tasks.workunit.client.0.smithi172.stderr:9 2024-05-08T16:13:15.991 INFO:tasks.workunit.client.0.smithi172.stderr:10' 2024-05-08T16:13:15.991 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-05-08T16:13:15.991 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T16:13:15.991 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-05-08T16:13:16.250 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836581 2024-05-08T16:13:16.250 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836581 2024-05-08T16:13:16.251 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836581' 2024-05-08T16:13:16.251 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T16:13:16.251 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-05-08T16:13:16.555 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673052 2024-05-08T16:13:16.555 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673052 2024-05-08T16:13:16.555 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836581 1-42949673052' 2024-05-08T16:13:16.556 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T16:13:16.556 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-05-08T16:13:16.952 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509526 2024-05-08T16:13:16.953 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509526 2024-05-08T16:13:16.953 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836581 1-42949673052 2-64424509526' 2024-05-08T16:13:16.953 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T16:13:16.953 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-05-08T16:13:17.124 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=326417514505 2024-05-08T16:13:17.124 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 326417514505 2024-05-08T16:13:17.125 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836581 1-42949673052 2-64424509526 3-326417514505' 2024-05-08T16:13:17.125 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T16:13:17.125 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-05-08T16:13:17.310 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182479 2024-05-08T16:13:17.310 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182479 2024-05-08T16:13:17.310 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836581 1-42949673052 2-64424509526 3-326417514505 4-107374182479' 2024-05-08T16:13:17.310 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T16:13:17.310 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-05-08T16:13:17.468 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=356482285571 2024-05-08T16:13:17.468 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 356482285571 2024-05-08T16:13:17.468 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836581 1-42949673052 2-64424509526 3-326417514505 4-107374182479 5-356482285571' 2024-05-08T16:13:17.469 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T16:13:17.469 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-05-08T16:13:17.619 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=150323855415 2024-05-08T16:13:17.619 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 150323855415 2024-05-08T16:13:17.619 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836581 1-42949673052 2-64424509526 3-326417514505 4-107374182479 5-356482285571 6-150323855415' 2024-05-08T16:13:17.620 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T16:13:17.620 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-05-08T16:13:17.771 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691887 2024-05-08T16:13:17.771 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691887 2024-05-08T16:13:17.771 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836581 1-42949673052 2-64424509526 3-326417514505 4-107374182479 5-356482285571 6-150323855415 7-171798691887' 2024-05-08T16:13:17.771 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T16:13:17.771 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-05-08T16:13:17.923 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528362 2024-05-08T16:13:17.924 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528362 2024-05-08T16:13:17.924 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836581 1-42949673052 2-64424509526 3-326417514505 4-107374182479 5-356482285571 6-150323855415 7-171798691887 8-193273528362' 2024-05-08T16:13:17.924 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T16:13:17.924 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-05-08T16:13:18.081 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=214748364833 2024-05-08T16:13:18.081 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 214748364833 2024-05-08T16:13:18.081 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836581 1-42949673052 2-64424509526 3-326417514505 4-107374182479 5-356482285571 6-150323855415 7-171798691887 8-193273528362 9-214748364833' 2024-05-08T16:13:18.081 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T16:13:18.081 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.10 flush_pg_stats 2024-05-08T16:13:18.234 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=236223201306 2024-05-08T16:13:18.234 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 236223201306 2024-05-08T16:13:18.234 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836581 1-42949673052 2-64424509526 3-326417514505 4-107374182479 5-356482285571 6-150323855415 7-171798691887 8-193273528362 9-214748364833 10-236223201306' 2024-05-08T16:13:18.234 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-08T16:13:18.235 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836581 2024-05-08T16:13:18.235 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-08T16:13:18.235 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-05-08T16:13:18.236 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836581 2024-05-08T16:13:18.236 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-08T16:13:18.237 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836581 2024-05-08T16:13:18.237 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836581' 2024-05-08T16:13:18.238 INFO:tasks.workunit.client.0.smithi172.stderr:waiting osd.0 seq 21474836581 2024-05-08T16:13:18.238 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-05-08T16:13:18.666 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836581 -lt 21474836581 2024-05-08T16:13:18.666 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-08T16:13:18.667 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673052 2024-05-08T16:13:18.667 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-08T16:13:18.668 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-05-08T16:13:18.669 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673052 2024-05-08T16:13:18.669 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-08T16:13:18.670 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673052 2024-05-08T16:13:18.670 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673052' 2024-05-08T16:13:18.670 INFO:tasks.workunit.client.0.smithi172.stderr:waiting osd.1 seq 42949673052 2024-05-08T16:13:18.671 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-05-08T16:13:19.104 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673052 -lt 42949673052 2024-05-08T16:13:19.104 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-08T16:13:19.105 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509526 2024-05-08T16:13:19.105 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-08T16:13:19.106 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-05-08T16:13:19.107 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509526 2024-05-08T16:13:19.107 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-08T16:13:19.109 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509526 2024-05-08T16:13:19.109 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509526' 2024-05-08T16:13:19.109 INFO:tasks.workunit.client.0.smithi172.stderr:waiting osd.2 seq 64424509526 2024-05-08T16:13:19.109 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-05-08T16:13:19.506 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509526 -lt 64424509526 2024-05-08T16:13:19.506 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-08T16:13:19.507 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-326417514505 2024-05-08T16:13:19.507 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-08T16:13:19.508 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-05-08T16:13:19.509 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-326417514505 2024-05-08T16:13:19.509 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-08T16:13:19.510 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=326417514505 2024-05-08T16:13:19.510 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 326417514505' 2024-05-08T16:13:19.510 INFO:tasks.workunit.client.0.smithi172.stderr:waiting osd.3 seq 326417514505 2024-05-08T16:13:19.510 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-05-08T16:13:19.944 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 326417514505 -lt 326417514505 2024-05-08T16:13:19.945 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-08T16:13:19.945 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182479 2024-05-08T16:13:19.945 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-08T16:13:19.947 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-05-08T16:13:19.948 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182479 2024-05-08T16:13:19.948 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-08T16:13:19.950 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182479 2024-05-08T16:13:19.950 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182479' 2024-05-08T16:13:19.950 INFO:tasks.workunit.client.0.smithi172.stderr:waiting osd.4 seq 107374182479 2024-05-08T16:13:19.950 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-05-08T16:13:20.383 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182479 -lt 107374182479 2024-05-08T16:13:20.383 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-08T16:13:20.383 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-356482285571 2024-05-08T16:13:20.383 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-08T16:13:20.384 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-05-08T16:13:20.385 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-356482285571 2024-05-08T16:13:20.385 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-08T16:13:20.386 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=356482285571 2024-05-08T16:13:20.386 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 356482285571' 2024-05-08T16:13:20.386 INFO:tasks.workunit.client.0.smithi172.stderr:waiting osd.5 seq 356482285571 2024-05-08T16:13:20.386 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-05-08T16:13:20.846 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 356482285572 -lt 356482285571 2024-05-08T16:13:20.846 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-08T16:13:20.847 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-150323855415 2024-05-08T16:13:20.847 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-08T16:13:20.848 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-05-08T16:13:20.849 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-150323855415 2024-05-08T16:13:20.849 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-08T16:13:20.850 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=150323855415 2024-05-08T16:13:20.850 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 150323855415' 2024-05-08T16:13:20.850 INFO:tasks.workunit.client.0.smithi172.stderr:waiting osd.6 seq 150323855415 2024-05-08T16:13:20.850 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-05-08T16:13:21.376 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 150323855415 -lt 150323855415 2024-05-08T16:13:21.376 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-08T16:13:21.377 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691887 2024-05-08T16:13:21.377 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-08T16:13:21.379 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-05-08T16:13:21.380 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691887 2024-05-08T16:13:21.380 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-08T16:13:21.381 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691887 2024-05-08T16:13:21.381 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691887' 2024-05-08T16:13:21.381 INFO:tasks.workunit.client.0.smithi172.stderr:waiting osd.7 seq 171798691887 2024-05-08T16:13:21.381 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-05-08T16:13:21.789 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691887 -lt 171798691887 2024-05-08T16:13:21.789 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-08T16:13:21.790 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-193273528362 2024-05-08T16:13:21.790 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-08T16:13:21.791 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-05-08T16:13:21.792 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528362 2024-05-08T16:13:21.792 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-08T16:13:21.794 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528362 2024-05-08T16:13:21.794 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528362' 2024-05-08T16:13:21.794 INFO:tasks.workunit.client.0.smithi172.stderr:waiting osd.8 seq 193273528362 2024-05-08T16:13:21.794 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-05-08T16:13:22.199 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528362 -lt 193273528362 2024-05-08T16:13:22.199 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-08T16:13:22.199 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-214748364833 2024-05-08T16:13:22.199 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-08T16:13:22.200 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-05-08T16:13:22.201 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-214748364833 2024-05-08T16:13:22.201 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-08T16:13:22.203 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=214748364833 2024-05-08T16:13:22.203 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 214748364833' 2024-05-08T16:13:22.203 INFO:tasks.workunit.client.0.smithi172.stderr:waiting osd.9 seq 214748364833 2024-05-08T16:13:22.203 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-05-08T16:13:22.631 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 214748364834 -lt 214748364833 2024-05-08T16:13:22.632 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-08T16:13:22.632 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 10-236223201306 2024-05-08T16:13:22.632 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-08T16:13:22.632 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=10 2024-05-08T16:13:22.633 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 10-236223201306 2024-05-08T16:13:22.634 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-08T16:13:22.636 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=236223201306 2024-05-08T16:13:22.636 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.10 seq 236223201306' 2024-05-08T16:13:22.636 INFO:tasks.workunit.client.0.smithi172.stderr:waiting osd.10 seq 236223201306 2024-05-08T16:13:22.636 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 10 2024-05-08T16:13:23.102 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 236223201307 -lt 236223201306 2024-05-08T16:13:23.103 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-05-08T16:13:23.103 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-05-08T16:13:23.103 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-05-08T16:13:23.622 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 16 == 0 2024-05-08T16:13:23.623 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-05-08T16:13:23.623 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-05-08T16:13:23.623 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-05-08T16:13:23.623 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-05-08T16:13:23.623 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-05-08T16:13:23.623 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-05-08T16:13:23.623 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-05-08T16:13:24.010 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=16 2024-05-08T16:13:24.010 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-05-08T16:13:24.011 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-05-08T16:13:24.011 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-05-08T16:13:24.539 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 16 = 16 2024-05-08T16:13:24.539 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-05-08T16:13:24.539 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-05-08T16:13:24.540 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:313: TEST_chunk_mapping: ceph osd erasure-code-profile set remap-profile plugin=lrc 'layers=[ [ "cDD", "" ] ]' mapping=_DD 'crush-steps=[ [ "choose", "osd", 0 ] ]' 2024-05-08T16:13:25.125 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:318: TEST_chunk_mapping: ceph osd erasure-code-profile get remap-profile 2024-05-08T16:13:25.573 INFO:tasks.workunit.client.0.smithi172.stdout:crush-device-class= 2024-05-08T16:13:25.573 INFO:tasks.workunit.client.0.smithi172.stdout:crush-failure-domain=host 2024-05-08T16:13:25.573 INFO:tasks.workunit.client.0.smithi172.stdout:crush-num-failure-domains=0 2024-05-08T16:13:25.573 INFO:tasks.workunit.client.0.smithi172.stdout:crush-osds-per-failure-domain=0 2024-05-08T16:13:25.573 INFO:tasks.workunit.client.0.smithi172.stdout:crush-root=default 2024-05-08T16:13:25.573 INFO:tasks.workunit.client.0.smithi172.stdout:crush-steps=[ [ "choose", "osd", 0 ] ] 2024-05-08T16:13:25.573 INFO:tasks.workunit.client.0.smithi172.stdout:k=-1 2024-05-08T16:13:25.573 INFO:tasks.workunit.client.0.smithi172.stdout:l=-1 2024-05-08T16:13:25.573 INFO:tasks.workunit.client.0.smithi172.stdout:layers=[ [ "cDD", "" ] ] 2024-05-08T16:13:25.574 INFO:tasks.workunit.client.0.smithi172.stdout:m=-1 2024-05-08T16:13:25.574 INFO:tasks.workunit.client.0.smithi172.stdout:mapping=_DD 2024-05-08T16:13:25.574 INFO:tasks.workunit.client.0.smithi172.stdout:plugin=lrc 2024-05-08T16:13:25.588 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:319: TEST_chunk_mapping: create_pool remap-pool 12 12 erasure remap-profile 2024-05-08T16:13:25.588 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create remap-pool 12 12 erasure remap-profile 2024-05-08T16:13:26.394 INFO:tasks.workunit.client.0.smithi172.stderr:pool 'remap-pool' already exists 2024-05-08T16:13:26.411 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-05-08T16:13:27.413 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:327: TEST_chunk_mapping: verify_chunk_mapping td/test-erasure-code remap-pool 1 2 2024-05-08T16:13:27.413 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:281: verify_chunk_mapping: local dir=td/test-erasure-code 2024-05-08T16:13:27.413 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:282: verify_chunk_mapping: local poolname=remap-pool 2024-05-08T16:13:27.413 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:283: verify_chunk_mapping: local first=1 2024-05-08T16:13:27.413 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:284: verify_chunk_mapping: local second=2 2024-05-08T16:13:27.414 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:286: verify_chunk_mapping: chunk_size 2024-05-08T16:13:27.415 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:270: chunk_size: ceph-conf --show-config-value osd_pool_erasure_code_stripe_unit 2024-05-08T16:13:27.438 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:270: chunk_size: echo 4096 2024-05-08T16:13:27.438 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:286: verify_chunk_mapping: printf '%*s' 4096 FIRSTremap-pool 2024-05-08T16:13:27.439 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:286: verify_chunk_mapping: chunk_size 2024-05-08T16:13:27.439 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:270: chunk_size: ceph-conf --show-config-value osd_pool_erasure_code_stripe_unit 2024-05-08T16:13:27.460 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:270: chunk_size: echo 4096 2024-05-08T16:13:27.460 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:286: verify_chunk_mapping: printf '%*s' 4096 SECONDremap-pool 2024-05-08T16:13:27.461 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:286: verify_chunk_mapping: local 'payload= FIRSTremap-pool SECONDremap-pool' 2024-05-08T16:13:27.461 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:287: verify_chunk_mapping: echo -n ' FIRSTremap-pool SECONDremap-pool' 2024-05-08T16:13:27.461 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:289: verify_chunk_mapping: rados --pool remap-pool put SOMETHINGremap-pool td/test-erasure-code/ORIGINAL 2024-05-08T16:13:29.711 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:290: verify_chunk_mapping: rados --pool remap-pool get SOMETHINGremap-pool td/test-erasure-code/COPY 2024-05-08T16:13:30.024 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:291: verify_chunk_mapping: get_osds remap-pool SOMETHINGremap-pool 2024-05-08T16:13:30.024 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=remap-pool 2024-05-08T16:13:30.025 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=SOMETHINGremap-pool 2024-05-08T16:13:30.025 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map remap-pool SOMETHINGremap-pool 2024-05-08T16:13:30.025 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-05-08T16:13:30.530 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=4 2024-05-08T16:13:30.530 INFO:tasks.workunit.client.0.smithi172.stderr:6 2024-05-08T16:13:30.531 INFO:tasks.workunit.client.0.smithi172.stderr:9' 2024-05-08T16:13:30.531 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 4 6 9 2024-05-08T16:13:30.531 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:291: verify_chunk_mapping: osds=('4' '6' '9') 2024-05-08T16:13:30.531 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:291: verify_chunk_mapping: local -a osds 2024-05-08T16:13:30.531 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:292: verify_chunk_mapping: (( i = 0 )) 2024-05-08T16:13:30.531 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:292: verify_chunk_mapping: (( i < 3 )) 2024-05-08T16:13:30.531 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:293: verify_chunk_mapping: ceph daemon osd.4 flush_journal 2024-05-08T16:13:30.659 INFO:tasks.workunit.client.0.smithi172.stderr:Can't get admin socket path: unable to get conf option admin_socket for osd: b"error parsing 'osd': expected string of the form TYPE.ID, valid types are: auth, mon, osd, mds, mgr, client\n" 2024-05-08T16:13:30.660 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:292: verify_chunk_mapping: (( i++ )) 2024-05-08T16:13:30.660 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:292: verify_chunk_mapping: (( i < 3 )) 2024-05-08T16:13:30.660 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:293: verify_chunk_mapping: ceph daemon osd.6 flush_journal 2024-05-08T16:13:30.787 INFO:tasks.workunit.client.0.smithi172.stderr:Can't get admin socket path: unable to get conf option admin_socket for osd: b"error parsing 'osd': expected string of the form TYPE.ID, valid types are: auth, mon, osd, mds, mgr, client\n" 2024-05-08T16:13:30.788 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:292: verify_chunk_mapping: (( i++ )) 2024-05-08T16:13:30.788 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:292: verify_chunk_mapping: (( i < 3 )) 2024-05-08T16:13:30.788 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:293: verify_chunk_mapping: ceph daemon osd.9 flush_journal 2024-05-08T16:13:30.925 INFO:tasks.workunit.client.0.smithi172.stderr:Can't get admin socket path: unable to get conf option admin_socket for osd: b"error parsing 'osd': expected string of the form TYPE.ID, valid types are: auth, mon, osd, mds, mgr, client\n" 2024-05-08T16:13:30.927 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:292: verify_chunk_mapping: (( i++ )) 2024-05-08T16:13:30.927 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:292: verify_chunk_mapping: (( i < 3 )) 2024-05-08T16:13:30.927 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:295: verify_chunk_mapping: diff td/test-erasure-code/ORIGINAL td/test-erasure-code/COPY 2024-05-08T16:13:30.928 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:296: verify_chunk_mapping: rm td/test-erasure-code/COPY 2024-05-08T16:13:30.930 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:298: verify_chunk_mapping: get_osds remap-pool SOMETHINGremap-pool 2024-05-08T16:13:30.931 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=remap-pool 2024-05-08T16:13:30.931 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=SOMETHINGremap-pool 2024-05-08T16:13:30.931 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map remap-pool SOMETHINGremap-pool 2024-05-08T16:13:30.931 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-05-08T16:13:31.345 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=4 2024-05-08T16:13:31.345 INFO:tasks.workunit.client.0.smithi172.stderr:6 2024-05-08T16:13:31.345 INFO:tasks.workunit.client.0.smithi172.stderr:9' 2024-05-08T16:13:31.345 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 4 6 9 2024-05-08T16:13:31.345 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:298: verify_chunk_mapping: osds=('4' '6' '9') 2024-05-08T16:13:31.345 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:298: verify_chunk_mapping: local -a osds 2024-05-08T16:13:31.347 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:299: verify_chunk_mapping: grep --quiet FIRSTremap-pool 2024-05-08T16:13:31.347 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:299: verify_chunk_mapping: objectstore_tool td/test-erasure-code 6 SOMETHINGremap-pool get-bytes 2024-05-08T16:13:31.347 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/test-erasure-code 2024-05-08T16:13:31.347 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-05-08T16:13:31.347 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=6 2024-05-08T16:13:31.348 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-05-08T16:13:31.348 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/test-erasure-code 6 SOMETHINGremap-pool get-bytes 2024-05-08T16:13:31.348 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/test-erasure-code 2024-05-08T16:13:31.348 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-05-08T16:13:31.348 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=6 2024-05-08T16:13:31.348 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-05-08T16:13:31.348 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/test-erasure-code TERM osd.6 2024-05-08T16:13:31.348 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-08T16:13:31.348 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-08T16:13:31.348 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-08T16:13:31.348 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-08T16:13:31.348 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-08T16:13:31.656 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-08T16:13:31.656 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/test-erasure-code 6 SOMETHINGremap-pool get-bytes 2024-05-08T16:13:31.656 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/test-erasure-code 2024-05-08T16:13:31.656 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-05-08T16:13:31.656 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=6 2024-05-08T16:13:31.656 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-05-08T16:13:31.656 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/test-erasure-code/6 2024-05-08T16:13:31.656 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/test-erasure-code/6 SOMETHINGremap-pool get-bytes 2024-05-08T16:13:35.320 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/test-erasure-code 6 2024-05-08T16:13:35.320 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/test-erasure-code 2024-05-08T16:13:35.320 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-05-08T16:13:35.320 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=6 2024-05-08T16:13:35.320 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-05-08T16:13:35.320 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/test-erasure-code/6 2024-05-08T16:13:35.320 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=f37874d1-6990-4767-8498-03a75cc31118 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false' 2024-05-08T16:13:35.320 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-08T16:13:35.320 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-05-08T16:13:35.320 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-08T16:13:35.321 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/test-erasure-code/6' 2024-05-08T16:13:35.321 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-code/6/journal' 2024-05-08T16:13:35.321 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-05-08T16:13:35.321 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-05-08T16:13:35.321 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/test-erasure-code' 2024-05-08T16:13:35.321 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-05-08T16:13:35.321 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-08T16:13:35.321 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-08T16:13:35.321 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-08T16:13:35.321 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T16:13:35.321 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.48735 2024-05-08T16:13:35.321 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.48735/$cluster-$name.asok' 2024-05-08T16:13:35.321 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.48735/$cluster-$name.asok' 2024-05-08T16:13:35.321 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-05-08T16:13:35.321 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/test-erasure-code/$name.log' 2024-05-08T16:13:35.322 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/test-erasure-code/$name.pid' 2024-05-08T16:13:35.322 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-08T16:13:35.322 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-08T16:13:35.322 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-08T16:13:35.322 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-08T16:13:35.322 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-05-08T16:13:35.322 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-05-08T16:13:35.322 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/test-erasure-code/6 2024-05-08T16:13:35.323 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.6 2024-05-08T16:13:35.323 INFO:tasks.workunit.client.0.smithi172.stderr:start osd.6 2024-05-08T16:13:35.323 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 6 --fsid=f37874d1-6990-4767-8498-03a75cc31118 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/6 --osd-journal=td/test-erasure-code/6/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.48735/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-05-08T16:13:35.323 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/test-erasure-code/6/whoami 2024-05-08T16:13:35.323 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 6 = 6 ']' 2024-05-08T16:13:35.323 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-05-08T16:13:35.323 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-05-08T16:13:35.324 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-05-08T16:13:35.354 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:13:35.363+0000 7f7306bad7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:13:35.356 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:13:35.367+0000 7f7306bad7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:13:35.358 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:13:35.367+0000 7f7306bad7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:13:35.743 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 6 2024-05-08T16:13:35.743 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-08T16:13:35.743 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=6 2024-05-08T16:13:35.743 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-08T16:13:35.743 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-08T16:13:35.743 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:13:35.743 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-08T16:13:35.743 INFO:tasks.workunit.client.0.smithi172.stderr:0 2024-05-08T16:13:35.743 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:13:35.743 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-05-08T16:13:35.917 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:13:35.927+0000 7f7306bad7c0 -1 Falling back to public interface 2024-05-08T16:13:36.195 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:13:37.197 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:13:37.197 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:13:37.197 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-08T16:13:37.197 INFO:tasks.workunit.client.0.smithi172.stderr:1 2024-05-08T16:13:37.197 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:13:37.197 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-05-08T16:13:37.598 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:13:38.600 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:13:38.600 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:13:38.600 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-08T16:13:38.600 INFO:tasks.workunit.client.0.smithi172.stderr:2 2024-05-08T16:13:38.601 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:13:38.601 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-05-08T16:13:38.644 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:13:38.655+0000 7f7306bad7c0 -1 osd.6 93 log_to_monitors true 2024-05-08T16:13:39.322 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:13:40.324 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:13:40.324 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:13:40.324 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-05-08T16:13:40.324 INFO:tasks.workunit.client.0.smithi172.stderr:3 2024-05-08T16:13:40.325 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:13:40.325 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-05-08T16:13:41.281 INFO:tasks.workunit.client.0.smithi172.stderr:osd.6 up in weight 1 up_from 98 up_thru 98 down_at 94 last_clean_interval [35,93) [v2:127.0.0.1:6850/1486889444,v1:127.0.0.1:6851/1486889444] [v2:127.0.0.1:6852/1486889444,v1:127.0.0.1:6853/1486889444] exists,up eea3e6ba-2b1f-4c9e-99f4-8290d432b42b 2024-05-08T16:13:41.282 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-08T16:13:41.282 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-08T16:13:41.282 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-08T16:13:41.282 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-05-08T16:13:41.282 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-05-08T16:13:41.282 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-05-08T16:13:41.282 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-05-08T16:13:41.282 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-05-08T16:13:41.283 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-05-08T16:13:41.283 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-05-08T16:13:41.283 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-05-08T16:13:41.283 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-05-08T16:13:41.283 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-05-08T16:13:41.393 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-05-08T16:13:41.393 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-05-08T16:13:41.393 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-05-08T16:13:41.393 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-05-08T16:13:41.393 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-05-08T16:13:41.394 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-05-08T16:13:42.104 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-05-08T16:13:42.104 INFO:tasks.workunit.client.0.smithi172.stderr:1 2024-05-08T16:13:42.104 INFO:tasks.workunit.client.0.smithi172.stderr:2 2024-05-08T16:13:42.104 INFO:tasks.workunit.client.0.smithi172.stderr:3 2024-05-08T16:13:42.104 INFO:tasks.workunit.client.0.smithi172.stderr:4 2024-05-08T16:13:42.104 INFO:tasks.workunit.client.0.smithi172.stderr:5 2024-05-08T16:13:42.104 INFO:tasks.workunit.client.0.smithi172.stderr:6 2024-05-08T16:13:42.105 INFO:tasks.workunit.client.0.smithi172.stderr:7 2024-05-08T16:13:42.105 INFO:tasks.workunit.client.0.smithi172.stderr:8 2024-05-08T16:13:42.105 INFO:tasks.workunit.client.0.smithi172.stderr:9 2024-05-08T16:13:42.105 INFO:tasks.workunit.client.0.smithi172.stderr:10' 2024-05-08T16:13:42.105 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-05-08T16:13:42.105 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T16:13:42.106 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-05-08T16:13:42.310 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836588 2024-05-08T16:13:42.310 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836588 2024-05-08T16:13:42.311 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836588' 2024-05-08T16:13:42.311 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T16:13:42.311 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-05-08T16:13:42.601 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673059 2024-05-08T16:13:42.602 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673059 2024-05-08T16:13:42.602 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836588 1-42949673059' 2024-05-08T16:13:42.602 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T16:13:42.602 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-05-08T16:13:42.759 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509533 2024-05-08T16:13:42.759 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509533 2024-05-08T16:13:42.759 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836588 1-42949673059 2-64424509533' 2024-05-08T16:13:42.759 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T16:13:42.759 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-05-08T16:13:42.911 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=326417514512 2024-05-08T16:13:42.911 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 326417514512 2024-05-08T16:13:42.911 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836588 1-42949673059 2-64424509533 3-326417514512' 2024-05-08T16:13:42.911 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T16:13:42.911 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-05-08T16:13:43.063 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182486 2024-05-08T16:13:43.063 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182486 2024-05-08T16:13:43.063 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836588 1-42949673059 2-64424509533 3-326417514512 4-107374182486' 2024-05-08T16:13:43.064 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T16:13:43.064 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-05-08T16:13:43.216 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=356482285578 2024-05-08T16:13:43.216 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 356482285578 2024-05-08T16:13:43.216 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836588 1-42949673059 2-64424509533 3-326417514512 4-107374182486 5-356482285578' 2024-05-08T16:13:43.216 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T16:13:43.216 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-05-08T16:13:43.366 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=420906795011 2024-05-08T16:13:43.366 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 420906795011 2024-05-08T16:13:43.366 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836588 1-42949673059 2-64424509533 3-326417514512 4-107374182486 5-356482285578 6-420906795011' 2024-05-08T16:13:43.366 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T16:13:43.367 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-05-08T16:13:43.509 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691894 2024-05-08T16:13:43.509 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691894 2024-05-08T16:13:43.509 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836588 1-42949673059 2-64424509533 3-326417514512 4-107374182486 5-356482285578 6-420906795011 7-171798691894' 2024-05-08T16:13:43.509 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T16:13:43.510 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-05-08T16:13:43.658 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528369 2024-05-08T16:13:43.658 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528369 2024-05-08T16:13:43.658 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836588 1-42949673059 2-64424509533 3-326417514512 4-107374182486 5-356482285578 6-420906795011 7-171798691894 8-193273528369' 2024-05-08T16:13:43.658 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T16:13:43.658 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-05-08T16:13:43.809 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=214748364840 2024-05-08T16:13:43.809 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 214748364840 2024-05-08T16:13:43.809 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836588 1-42949673059 2-64424509533 3-326417514512 4-107374182486 5-356482285578 6-420906795011 7-171798691894 8-193273528369 9-214748364840' 2024-05-08T16:13:43.809 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T16:13:43.809 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.10 flush_pg_stats 2024-05-08T16:13:43.962 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=236223201314 2024-05-08T16:13:43.962 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 236223201314 2024-05-08T16:13:43.962 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836588 1-42949673059 2-64424509533 3-326417514512 4-107374182486 5-356482285578 6-420906795011 7-171798691894 8-193273528369 9-214748364840 10-236223201314' 2024-05-08T16:13:43.963 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-08T16:13:43.963 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836588 2024-05-08T16:13:43.963 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-08T16:13:43.963 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-05-08T16:13:43.963 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836588 2024-05-08T16:13:43.963 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-08T16:13:43.963 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836588 2024-05-08T16:13:43.963 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836588' 2024-05-08T16:13:43.963 INFO:tasks.workunit.client.0.smithi172.stderr:waiting osd.0 seq 21474836588 2024-05-08T16:13:43.963 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-05-08T16:13:44.423 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836588 -lt 21474836588 2024-05-08T16:13:44.423 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-08T16:13:44.424 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673059 2024-05-08T16:13:44.424 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-08T16:13:44.425 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-05-08T16:13:44.425 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673059 2024-05-08T16:13:44.426 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-08T16:13:44.427 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673059 2024-05-08T16:13:44.427 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673059' 2024-05-08T16:13:44.427 INFO:tasks.workunit.client.0.smithi172.stderr:waiting osd.1 seq 42949673059 2024-05-08T16:13:44.427 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-05-08T16:13:44.841 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673059 -lt 42949673059 2024-05-08T16:13:44.842 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-08T16:13:44.842 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509533 2024-05-08T16:13:44.842 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-08T16:13:44.844 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-05-08T16:13:44.845 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509533 2024-05-08T16:13:44.845 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-08T16:13:44.846 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509533 2024-05-08T16:13:44.846 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509533' 2024-05-08T16:13:44.847 INFO:tasks.workunit.client.0.smithi172.stderr:waiting osd.2 seq 64424509533 2024-05-08T16:13:44.847 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-05-08T16:13:45.291 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509534 -lt 64424509533 2024-05-08T16:13:45.291 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-08T16:13:45.291 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-326417514512 2024-05-08T16:13:45.291 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-08T16:13:45.293 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-05-08T16:13:45.294 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-326417514512 2024-05-08T16:13:45.294 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-08T16:13:45.295 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=326417514512 2024-05-08T16:13:45.296 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 326417514512' 2024-05-08T16:13:45.296 INFO:tasks.workunit.client.0.smithi172.stderr:waiting osd.3 seq 326417514512 2024-05-08T16:13:45.296 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-05-08T16:13:45.699 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 326417514512 -lt 326417514512 2024-05-08T16:13:45.699 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-08T16:13:45.699 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-08T16:13:45.699 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182486 2024-05-08T16:13:45.700 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-05-08T16:13:45.701 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182486 2024-05-08T16:13:45.701 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-08T16:13:45.702 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182486 2024-05-08T16:13:45.702 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182486' 2024-05-08T16:13:45.702 INFO:tasks.workunit.client.0.smithi172.stderr:waiting osd.4 seq 107374182486 2024-05-08T16:13:45.703 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-05-08T16:13:46.135 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182487 -lt 107374182486 2024-05-08T16:13:46.135 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-08T16:13:46.135 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-356482285578 2024-05-08T16:13:46.136 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-08T16:13:46.136 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-05-08T16:13:46.137 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-356482285578 2024-05-08T16:13:46.137 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-08T16:13:46.139 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=356482285578 2024-05-08T16:13:46.139 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 356482285578' 2024-05-08T16:13:46.139 INFO:tasks.workunit.client.0.smithi172.stderr:waiting osd.5 seq 356482285578 2024-05-08T16:13:46.139 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-05-08T16:13:46.550 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 356482285579 -lt 356482285578 2024-05-08T16:13:46.550 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-08T16:13:46.551 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-420906795011 2024-05-08T16:13:46.551 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-08T16:13:46.552 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-05-08T16:13:46.553 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-420906795011 2024-05-08T16:13:46.553 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-08T16:13:46.554 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=420906795011 2024-05-08T16:13:46.554 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 420906795011' 2024-05-08T16:13:46.554 INFO:tasks.workunit.client.0.smithi172.stderr:waiting osd.6 seq 420906795011 2024-05-08T16:13:46.554 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-05-08T16:13:46.980 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 420906795012 -lt 420906795011 2024-05-08T16:13:46.981 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-08T16:13:46.981 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691894 2024-05-08T16:13:46.981 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-08T16:13:46.981 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-05-08T16:13:46.981 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691894 2024-05-08T16:13:46.981 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-08T16:13:46.982 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691894 2024-05-08T16:13:46.982 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691894' 2024-05-08T16:13:46.982 INFO:tasks.workunit.client.0.smithi172.stderr:waiting osd.7 seq 171798691894 2024-05-08T16:13:46.982 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-05-08T16:13:47.421 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691895 -lt 171798691894 2024-05-08T16:13:47.421 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-08T16:13:47.422 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-08T16:13:47.422 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-193273528369 2024-05-08T16:13:47.423 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-05-08T16:13:47.423 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528369 2024-05-08T16:13:47.424 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-08T16:13:47.425 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528369 2024-05-08T16:13:47.426 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528369' 2024-05-08T16:13:47.426 INFO:tasks.workunit.client.0.smithi172.stderr:waiting osd.8 seq 193273528369 2024-05-08T16:13:47.426 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-05-08T16:13:47.856 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528370 -lt 193273528369 2024-05-08T16:13:47.856 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-08T16:13:47.857 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-08T16:13:47.857 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-214748364840 2024-05-08T16:13:47.858 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-05-08T16:13:47.859 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-214748364840 2024-05-08T16:13:47.859 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-08T16:13:47.860 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=214748364840 2024-05-08T16:13:47.860 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 214748364840' 2024-05-08T16:13:47.861 INFO:tasks.workunit.client.0.smithi172.stderr:waiting osd.9 seq 214748364840 2024-05-08T16:13:47.861 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-05-08T16:13:48.297 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 214748364841 -lt 214748364840 2024-05-08T16:13:48.297 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-08T16:13:48.298 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 10-236223201314 2024-05-08T16:13:48.298 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-08T16:13:48.300 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=10 2024-05-08T16:13:48.300 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 10-236223201314 2024-05-08T16:13:48.301 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-08T16:13:48.302 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=236223201314 2024-05-08T16:13:48.302 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.10 seq 236223201314' 2024-05-08T16:13:48.302 INFO:tasks.workunit.client.0.smithi172.stderr:waiting osd.10 seq 236223201314 2024-05-08T16:13:48.302 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 10 2024-05-08T16:13:48.778 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 236223201314 -lt 236223201314 2024-05-08T16:13:48.779 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-05-08T16:13:48.779 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-05-08T16:13:48.779 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-05-08T16:13:49.341 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 28 == 0 2024-05-08T16:13:49.341 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-05-08T16:13:49.341 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-05-08T16:13:49.341 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-05-08T16:13:49.341 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-05-08T16:13:49.341 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-05-08T16:13:49.341 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-05-08T16:13:49.342 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-05-08T16:13:49.784 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=28 2024-05-08T16:13:49.784 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-05-08T16:13:49.784 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-05-08T16:13:49.784 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-05-08T16:13:50.307 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 28 = 28 2024-05-08T16:13:50.307 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-05-08T16:13:50.307 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-05-08T16:13:50.310 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:300: verify_chunk_mapping: objectstore_tool td/test-erasure-code 9 SOMETHINGremap-pool get-bytes 2024-05-08T16:13:50.310 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/test-erasure-code 2024-05-08T16:13:50.310 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-05-08T16:13:50.310 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=9 2024-05-08T16:13:50.310 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-05-08T16:13:50.310 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/test-erasure-code 9 SOMETHINGremap-pool get-bytes 2024-05-08T16:13:50.310 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/test-erasure-code 2024-05-08T16:13:50.310 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-05-08T16:13:50.311 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=9 2024-05-08T16:13:50.311 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-05-08T16:13:50.311 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/test-erasure-code TERM osd.9 2024-05-08T16:13:50.311 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:300: verify_chunk_mapping: grep --quiet SECONDremap-pool 2024-05-08T16:13:50.311 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-08T16:13:50.311 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-08T16:13:50.311 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-08T16:13:50.311 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-08T16:13:50.311 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-08T16:13:50.618 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-08T16:13:50.618 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/test-erasure-code 9 SOMETHINGremap-pool get-bytes 2024-05-08T16:13:50.618 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/test-erasure-code 2024-05-08T16:13:50.618 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-05-08T16:13:50.618 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=9 2024-05-08T16:13:50.618 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-05-08T16:13:50.618 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/test-erasure-code/9 2024-05-08T16:13:50.618 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/test-erasure-code/9 SOMETHINGremap-pool get-bytes 2024-05-08T16:13:53.964 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/test-erasure-code 9 2024-05-08T16:13:53.965 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/test-erasure-code 2024-05-08T16:13:53.965 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-05-08T16:13:53.965 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=9 2024-05-08T16:13:53.965 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-05-08T16:13:53.965 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/test-erasure-code/9 2024-05-08T16:13:53.965 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=f37874d1-6990-4767-8498-03a75cc31118 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false' 2024-05-08T16:13:53.965 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-08T16:13:53.965 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-05-08T16:13:53.965 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-08T16:13:53.965 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/test-erasure-code/9' 2024-05-08T16:13:53.965 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-code/9/journal' 2024-05-08T16:13:53.965 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-05-08T16:13:53.966 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-05-08T16:13:53.966 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/test-erasure-code' 2024-05-08T16:13:53.966 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-05-08T16:13:53.966 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-08T16:13:53.966 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-08T16:13:53.966 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-08T16:13:53.966 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T16:13:53.966 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.48735 2024-05-08T16:13:53.966 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.48735/$cluster-$name.asok' 2024-05-08T16:13:53.968 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.48735/$cluster-$name.asok' 2024-05-08T16:13:53.968 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-05-08T16:13:53.968 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/test-erasure-code/$name.log' 2024-05-08T16:13:53.968 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/test-erasure-code/$name.pid' 2024-05-08T16:13:53.969 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-08T16:13:53.969 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-08T16:13:53.969 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-08T16:13:53.969 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-08T16:13:53.969 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-05-08T16:13:53.969 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-05-08T16:13:53.969 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/test-erasure-code/9 2024-05-08T16:13:53.969 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.9 2024-05-08T16:13:53.969 INFO:tasks.workunit.client.0.smithi172.stderr:start osd.9 2024-05-08T16:13:53.969 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 9 --fsid=f37874d1-6990-4767-8498-03a75cc31118 --auth-supported=none --mon-host=127.0.0.1:7101 --mon-osd-prime-pg-temp=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-code/9 --osd-journal=td/test-erasure-code/9/journal --chdir= --run-dir=td/test-erasure-code '--admin-socket=/tmp/ceph-asok.48735/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-code/$name.log' '--pid-file=td/test-erasure-code/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-05-08T16:13:53.969 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/test-erasure-code/9/whoami 2024-05-08T16:13:53.970 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 9 = 9 ']' 2024-05-08T16:13:53.970 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-05-08T16:13:53.970 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-05-08T16:13:53.970 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-05-08T16:13:54.007 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:13:54.014+0000 7fa6506027c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:13:54.009 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:13:54.018+0000 7fa6506027c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:13:54.010 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:13:54.018+0000 7fa6506027c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:13:54.401 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 9 2024-05-08T16:13:54.402 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-08T16:13:54.402 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=9 2024-05-08T16:13:54.402 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-08T16:13:54.402 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-08T16:13:54.402 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:13:54.402 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-08T16:13:54.402 INFO:tasks.workunit.client.0.smithi172.stderr:0 2024-05-08T16:13:54.402 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-05-08T16:13:54.402 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:13:54.573 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:13:54.582+0000 7fa6506027c0 -1 Falling back to public interface 2024-05-08T16:13:54.858 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:13:55.860 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:13:55.860 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:13:55.860 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-08T16:13:55.860 INFO:tasks.workunit.client.0.smithi172.stderr:1 2024-05-08T16:13:55.861 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:13:55.861 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-05-08T16:13:56.288 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:13:57.290 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:13:57.290 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:13:57.290 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-08T16:13:57.290 INFO:tasks.workunit.client.0.smithi172.stderr:2 2024-05-08T16:13:57.290 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:13:57.290 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-05-08T16:13:57.314 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:13:57.322+0000 7fa6506027c0 -1 osd.9 100 log_to_monitors true 2024-05-08T16:13:57.754 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:13:58.756 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:13:58.756 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:13:58.756 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-05-08T16:13:58.756 INFO:tasks.workunit.client.0.smithi172.stderr:3 2024-05-08T16:13:58.756 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:13:58.756 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.9 up' 2024-05-08T16:13:59.402 INFO:tasks.workunit.client.0.smithi172.stderr:osd.9 up in weight 1 up_from 105 up_thru 105 down_at 101 last_clean_interval [50,100) [v2:127.0.0.1:6874/762776500,v1:127.0.0.1:6875/762776500] [v2:127.0.0.1:6876/762776500,v1:127.0.0.1:6877/762776500] exists,up 54514676-7fc6-4e04-8a2c-b3fc55239502 2024-05-08T16:13:59.402 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-08T16:13:59.402 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-08T16:13:59.403 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-08T16:13:59.403 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-05-08T16:13:59.403 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-05-08T16:13:59.403 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-05-08T16:13:59.403 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-05-08T16:13:59.403 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-05-08T16:13:59.404 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-05-08T16:13:59.404 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-05-08T16:13:59.404 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-05-08T16:13:59.404 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-05-08T16:13:59.404 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-05-08T16:13:59.516 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-05-08T16:13:59.516 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-05-08T16:13:59.516 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-05-08T16:13:59.516 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-05-08T16:13:59.516 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-05-08T16:13:59.516 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-05-08T16:14:00.996 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-05-08T16:14:00.996 INFO:tasks.workunit.client.0.smithi172.stderr:1 2024-05-08T16:14:00.996 INFO:tasks.workunit.client.0.smithi172.stderr:2 2024-05-08T16:14:00.996 INFO:tasks.workunit.client.0.smithi172.stderr:3 2024-05-08T16:14:00.996 INFO:tasks.workunit.client.0.smithi172.stderr:4 2024-05-08T16:14:00.996 INFO:tasks.workunit.client.0.smithi172.stderr:5 2024-05-08T16:14:00.996 INFO:tasks.workunit.client.0.smithi172.stderr:6 2024-05-08T16:14:00.996 INFO:tasks.workunit.client.0.smithi172.stderr:7 2024-05-08T16:14:00.996 INFO:tasks.workunit.client.0.smithi172.stderr:8 2024-05-08T16:14:00.996 INFO:tasks.workunit.client.0.smithi172.stderr:9 2024-05-08T16:14:00.996 INFO:tasks.workunit.client.0.smithi172.stderr:10' 2024-05-08T16:14:00.997 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-05-08T16:14:00.997 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T16:14:00.997 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-05-08T16:14:01.197 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836594 2024-05-08T16:14:01.197 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836594 2024-05-08T16:14:01.197 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836594' 2024-05-08T16:14:01.197 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T16:14:01.197 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-05-08T16:14:01.496 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673065 2024-05-08T16:14:01.496 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673065 2024-05-08T16:14:01.496 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836594 1-42949673065' 2024-05-08T16:14:01.496 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T16:14:01.496 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-05-08T16:14:01.651 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509539 2024-05-08T16:14:01.651 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509539 2024-05-08T16:14:01.651 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836594 1-42949673065 2-64424509539' 2024-05-08T16:14:01.651 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T16:14:01.651 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-05-08T16:14:01.806 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=326417514518 2024-05-08T16:14:01.807 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 326417514518 2024-05-08T16:14:01.807 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836594 1-42949673065 2-64424509539 3-326417514518' 2024-05-08T16:14:01.807 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T16:14:01.807 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-05-08T16:14:01.955 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182492 2024-05-08T16:14:01.955 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182492 2024-05-08T16:14:01.955 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836594 1-42949673065 2-64424509539 3-326417514518 4-107374182492' 2024-05-08T16:14:01.955 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T16:14:01.955 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-05-08T16:14:02.104 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=356482285584 2024-05-08T16:14:02.104 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 356482285584 2024-05-08T16:14:02.104 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836594 1-42949673065 2-64424509539 3-326417514518 4-107374182492 5-356482285584' 2024-05-08T16:14:02.104 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T16:14:02.104 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-05-08T16:14:02.255 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=420906795017 2024-05-08T16:14:02.256 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 420906795017 2024-05-08T16:14:02.256 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836594 1-42949673065 2-64424509539 3-326417514518 4-107374182492 5-356482285584 6-420906795017' 2024-05-08T16:14:02.256 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T16:14:02.256 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-05-08T16:14:02.410 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691900 2024-05-08T16:14:02.410 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691900 2024-05-08T16:14:02.410 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836594 1-42949673065 2-64424509539 3-326417514518 4-107374182492 5-356482285584 6-420906795017 7-171798691900' 2024-05-08T16:14:02.410 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T16:14:02.410 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-05-08T16:14:02.568 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528375 2024-05-08T16:14:02.569 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528375 2024-05-08T16:14:02.569 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836594 1-42949673065 2-64424509539 3-326417514518 4-107374182492 5-356482285584 6-420906795017 7-171798691900 8-193273528375' 2024-05-08T16:14:02.569 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T16:14:02.569 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-05-08T16:14:02.752 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=450971566083 2024-05-08T16:14:02.752 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 450971566083 2024-05-08T16:14:02.752 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836594 1-42949673065 2-64424509539 3-326417514518 4-107374182492 5-356482285584 6-420906795017 7-171798691900 8-193273528375 9-450971566083' 2024-05-08T16:14:02.752 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T16:14:02.752 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.10 flush_pg_stats 2024-05-08T16:14:02.904 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=236223201319 2024-05-08T16:14:02.904 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 236223201319 2024-05-08T16:14:02.904 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836594 1-42949673065 2-64424509539 3-326417514518 4-107374182492 5-356482285584 6-420906795017 7-171798691900 8-193273528375 9-450971566083 10-236223201319' 2024-05-08T16:14:02.904 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-08T16:14:02.904 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836594 2024-05-08T16:14:02.904 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-08T16:14:02.905 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-05-08T16:14:02.906 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836594 2024-05-08T16:14:02.906 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-08T16:14:02.907 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836594 2024-05-08T16:14:02.907 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836594' 2024-05-08T16:14:02.907 INFO:tasks.workunit.client.0.smithi172.stderr:waiting osd.0 seq 21474836594 2024-05-08T16:14:02.908 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-05-08T16:14:03.317 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836594 -lt 21474836594 2024-05-08T16:14:03.317 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-08T16:14:03.318 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673065 2024-05-08T16:14:03.318 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-08T16:14:03.320 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-05-08T16:14:03.320 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673065 2024-05-08T16:14:03.320 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-08T16:14:03.321 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673065 2024-05-08T16:14:03.322 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673065' 2024-05-08T16:14:03.322 INFO:tasks.workunit.client.0.smithi172.stderr:waiting osd.1 seq 42949673065 2024-05-08T16:14:03.322 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-05-08T16:14:03.755 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673065 -lt 42949673065 2024-05-08T16:14:03.755 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-08T16:14:03.755 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509539 2024-05-08T16:14:03.756 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-08T16:14:03.757 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-05-08T16:14:03.758 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509539 2024-05-08T16:14:03.758 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-08T16:14:03.759 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509539 2024-05-08T16:14:03.759 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509539' 2024-05-08T16:14:03.759 INFO:tasks.workunit.client.0.smithi172.stderr:waiting osd.2 seq 64424509539 2024-05-08T16:14:03.759 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-05-08T16:14:04.154 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509539 -lt 64424509539 2024-05-08T16:14:04.154 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-08T16:14:04.154 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-326417514518 2024-05-08T16:14:04.155 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-08T16:14:04.156 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-05-08T16:14:04.157 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-326417514518 2024-05-08T16:14:04.157 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-08T16:14:04.158 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=326417514518 2024-05-08T16:14:04.158 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 326417514518' 2024-05-08T16:14:04.159 INFO:tasks.workunit.client.0.smithi172.stderr:waiting osd.3 seq 326417514518 2024-05-08T16:14:04.159 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-05-08T16:14:04.788 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 326417514518 -lt 326417514518 2024-05-08T16:14:04.788 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-08T16:14:04.789 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-08T16:14:04.789 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182492 2024-05-08T16:14:04.790 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-05-08T16:14:04.790 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182492 2024-05-08T16:14:04.791 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-08T16:14:04.792 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182492 2024-05-08T16:14:04.792 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182492' 2024-05-08T16:14:04.792 INFO:tasks.workunit.client.0.smithi172.stderr:waiting osd.4 seq 107374182492 2024-05-08T16:14:04.792 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-05-08T16:14:05.214 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182493 -lt 107374182492 2024-05-08T16:14:05.214 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-08T16:14:05.215 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-356482285584 2024-05-08T16:14:05.215 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-08T16:14:05.216 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-05-08T16:14:05.216 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-356482285584 2024-05-08T16:14:05.216 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-08T16:14:05.217 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=356482285584 2024-05-08T16:14:05.218 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 356482285584' 2024-05-08T16:14:05.218 INFO:tasks.workunit.client.0.smithi172.stderr:waiting osd.5 seq 356482285584 2024-05-08T16:14:05.218 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-05-08T16:14:05.622 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 356482285585 -lt 356482285584 2024-05-08T16:14:05.622 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-08T16:14:05.622 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-420906795017 2024-05-08T16:14:05.622 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-08T16:14:05.624 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-05-08T16:14:05.624 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-420906795017 2024-05-08T16:14:05.624 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-08T16:14:05.625 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=420906795017 2024-05-08T16:14:05.626 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 420906795017' 2024-05-08T16:14:05.626 INFO:tasks.workunit.client.0.smithi172.stderr:waiting osd.6 seq 420906795017 2024-05-08T16:14:05.626 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-05-08T16:14:06.070 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 420906795018 -lt 420906795017 2024-05-08T16:14:06.071 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-08T16:14:06.071 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691900 2024-05-08T16:14:06.072 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-08T16:14:06.073 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-05-08T16:14:06.073 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691900 2024-05-08T16:14:06.073 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-08T16:14:06.075 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691900 2024-05-08T16:14:06.075 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691900' 2024-05-08T16:14:06.075 INFO:tasks.workunit.client.0.smithi172.stderr:waiting osd.7 seq 171798691900 2024-05-08T16:14:06.075 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-05-08T16:14:06.485 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691900 -lt 171798691900 2024-05-08T16:14:06.485 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-08T16:14:06.486 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-193273528375 2024-05-08T16:14:06.486 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-08T16:14:06.487 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-05-08T16:14:06.487 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528375 2024-05-08T16:14:06.487 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-08T16:14:06.489 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528375 2024-05-08T16:14:06.489 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528375' 2024-05-08T16:14:06.489 INFO:tasks.workunit.client.0.smithi172.stderr:waiting osd.8 seq 193273528375 2024-05-08T16:14:06.489 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-05-08T16:14:06.900 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528376 -lt 193273528375 2024-05-08T16:14:06.900 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-08T16:14:06.901 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-450971566083 2024-05-08T16:14:06.901 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-08T16:14:06.902 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-05-08T16:14:06.903 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-450971566083 2024-05-08T16:14:06.903 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-08T16:14:06.904 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=450971566083 2024-05-08T16:14:06.904 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 450971566083' 2024-05-08T16:14:06.904 INFO:tasks.workunit.client.0.smithi172.stderr:waiting osd.9 seq 450971566083 2024-05-08T16:14:06.904 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-05-08T16:14:07.319 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 450971566084 -lt 450971566083 2024-05-08T16:14:07.320 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-08T16:14:07.320 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 10-236223201319 2024-05-08T16:14:07.320 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-08T16:14:07.321 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=10 2024-05-08T16:14:07.322 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 10-236223201319 2024-05-08T16:14:07.322 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-08T16:14:07.323 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=236223201319 2024-05-08T16:14:07.323 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.10 seq 236223201319' 2024-05-08T16:14:07.323 INFO:tasks.workunit.client.0.smithi172.stderr:waiting osd.10 seq 236223201319 2024-05-08T16:14:07.323 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 10 2024-05-08T16:14:07.744 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 236223201320 -lt 236223201319 2024-05-08T16:14:07.744 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-05-08T16:14:07.745 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-05-08T16:14:07.745 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-05-08T16:14:08.292 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 28 == 0 2024-05-08T16:14:08.292 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-05-08T16:14:08.292 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-05-08T16:14:08.292 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-05-08T16:14:08.292 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-05-08T16:14:08.292 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-05-08T16:14:08.292 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-05-08T16:14:08.293 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-05-08T16:14:08.679 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=28 2024-05-08T16:14:08.679 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-05-08T16:14:08.679 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-05-08T16:14:08.679 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-05-08T16:14:09.198 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 28 = 28 2024-05-08T16:14:09.199 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-05-08T16:14:09.199 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-05-08T16:14:09.199 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:329: TEST_chunk_mapping: delete_pool remap-pool 2024-05-08T16:14:09.199 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=remap-pool 2024-05-08T16:14:09.199 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete remap-pool remap-pool --yes-i-really-really-mean-it 2024-05-08T16:14:09.816 INFO:tasks.workunit.client.0.smithi172.stderr:pool 'remap-pool' does not exist 2024-05-08T16:14:09.838 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:330: TEST_chunk_mapping: ceph osd erasure-code-profile rm remap-profile 2024-05-08T16:14:10.607 INFO:tasks.workunit.client.0.smithi172.stderr:erasure-code-profile remap-profile does not exist 2024-05-08T16:14:10.626 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:47: run: for func in $funcs 2024-05-08T16:14:10.627 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:48: run: TEST_rados_put_get_isa td/test-erasure-code 2024-05-08T16:14:10.627 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:193: TEST_rados_put_get_isa: erasure_code_plugin_exists isa 2024-05-08T16:14:10.627 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2051: erasure_code_plugin_exists: local plugin=isa 2024-05-08T16:14:10.627 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2052: erasure_code_plugin_exists: local status 2024-05-08T16:14:10.627 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2053: erasure_code_plugin_exists: local grepstr 2024-05-08T16:14:10.627 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2054: erasure_code_plugin_exists: local s 2024-05-08T16:14:10.627 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2055: erasure_code_plugin_exists: case `uname` in 2024-05-08T16:14:10.628 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2055: erasure_code_plugin_exists: uname 2024-05-08T16:14:10.628 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2057: erasure_code_plugin_exists: grepstr='isa.*No such file' 2024-05-08T16:14:10.629 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2060: erasure_code_plugin_exists: ceph osd erasure-code-profile set TESTPROFILE plugin=isa 2024-05-08T16:14:11.858 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2060: erasure_code_plugin_exists: s= 2024-05-08T16:14:11.858 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2061: erasure_code_plugin_exists: local status=0 2024-05-08T16:14:11.858 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2062: erasure_code_plugin_exists: '[' 0 -eq 0 ']' 2024-05-08T16:14:11.858 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2063: erasure_code_plugin_exists: ceph osd erasure-code-profile rm TESTPROFILE 2024-05-08T16:14:12.756 INFO:tasks.workunit.client.0.smithi172.stderr:erasure-code-profile TESTPROFILE does not exist 2024-05-08T16:14:12.773 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: erasure_code_plugin_exists: return 0 2024-05-08T16:14:12.773 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:197: TEST_rados_put_get_isa: local dir=td/test-erasure-code 2024-05-08T16:14:12.773 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:198: TEST_rados_put_get_isa: local poolname=pool-isa 2024-05-08T16:14:12.773 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:200: TEST_rados_put_get_isa: ceph osd erasure-code-profile set profile-isa plugin=isa crush-failure-domain=osd 2024-05-08T16:14:13.837 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:203: TEST_rados_put_get_isa: create_pool pool-isa 1 1 erasure profile-isa 2024-05-08T16:14:13.837 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create pool-isa 1 1 erasure profile-isa 2024-05-08T16:14:15.049 INFO:tasks.workunit.client.0.smithi172.stderr:pool 'pool-isa' already exists 2024-05-08T16:14:15.066 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-05-08T16:14:16.069 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:206: TEST_rados_put_get_isa: rados_put_get td/test-erasure-code pool-isa 2024-05-08T16:14:16.069 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:66: rados_put_get: local dir=td/test-erasure-code 2024-05-08T16:14:16.069 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:67: rados_put_get: local poolname=pool-isa 2024-05-08T16:14:16.069 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:68: rados_put_get: local objname=SOMETHING 2024-05-08T16:14:16.069 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2024-05-08T16:14:16.069 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 AAA 2024-05-08T16:14:16.069 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2024-05-08T16:14:16.070 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 BBB 2024-05-08T16:14:16.070 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2024-05-08T16:14:16.070 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 CCCC 2024-05-08T16:14:16.070 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2024-05-08T16:14:16.070 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 DDDD 2024-05-08T16:14:16.070 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:78: rados_put_get: rados --pool pool-isa put SOMETHING td/test-erasure-code/ORIGINAL 2024-05-08T16:14:17.358 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:79: rados_put_get: rados --pool pool-isa get SOMETHING td/test-erasure-code/COPY 2024-05-08T16:14:17.568 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:80: rados_put_get: diff td/test-erasure-code/ORIGINAL td/test-erasure-code/COPY 2024-05-08T16:14:17.570 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:81: rados_put_get: rm td/test-erasure-code/COPY 2024-05-08T16:14:17.573 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:88: rados_put_get: get_osds pool-isa SOMETHING 2024-05-08T16:14:17.574 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-isa 2024-05-08T16:14:17.574 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=SOMETHING 2024-05-08T16:14:17.575 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-isa SOMETHING 2024-05-08T16:14:17.575 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-05-08T16:14:18.122 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=4 2024-05-08T16:14:18.122 INFO:tasks.workunit.client.0.smithi172.stderr:10 2024-05-08T16:14:18.122 INFO:tasks.workunit.client.0.smithi172.stderr:0 2024-05-08T16:14:18.122 INFO:tasks.workunit.client.0.smithi172.stderr:7 2024-05-08T16:14:18.122 INFO:tasks.workunit.client.0.smithi172.stderr:6 2024-05-08T16:14:18.122 INFO:tasks.workunit.client.0.smithi172.stderr:8 2024-05-08T16:14:18.122 INFO:tasks.workunit.client.0.smithi172.stderr:2 2024-05-08T16:14:18.122 INFO:tasks.workunit.client.0.smithi172.stderr:1 2024-05-08T16:14:18.122 INFO:tasks.workunit.client.0.smithi172.stderr:5 2024-05-08T16:14:18.122 INFO:tasks.workunit.client.0.smithi172.stderr:9' 2024-05-08T16:14:18.122 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 4 10 0 7 6 8 2 1 5 9 2024-05-08T16:14:18.123 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:88: rados_put_get: initial_osds=('4' '10' '0' '7' '6' '8' '2' '1' '5' '9') 2024-05-08T16:14:18.123 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:88: rados_put_get: local -a initial_osds 2024-05-08T16:14:18.123 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:89: rados_put_get: local last=9 2024-05-08T16:14:18.123 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:90: rados_put_get: ceph osd out 9 2024-05-08T16:14:18.680 INFO:tasks.workunit.client.0.smithi172.stderr:osd.9 is already out. 2024-05-08T16:14:18.696 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:93: rados_put_get: sleep 5 2024-05-08T16:14:23.699 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:94: rados_put_get: get_osds pool-isa SOMETHING 2024-05-08T16:14:23.699 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-isa 2024-05-08T16:14:23.699 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=SOMETHING 2024-05-08T16:14:23.700 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:94: rados_put_get: grep '\<9\>' 2024-05-08T16:14:23.700 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-isa SOMETHING 2024-05-08T16:14:23.701 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-05-08T16:14:24.119 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=4 2024-05-08T16:14:24.120 INFO:tasks.workunit.client.0.smithi172.stderr:10 2024-05-08T16:14:24.120 INFO:tasks.workunit.client.0.smithi172.stderr:0 2024-05-08T16:14:24.120 INFO:tasks.workunit.client.0.smithi172.stderr:7 2024-05-08T16:14:24.120 INFO:tasks.workunit.client.0.smithi172.stderr:6 2024-05-08T16:14:24.120 INFO:tasks.workunit.client.0.smithi172.stderr:8 2024-05-08T16:14:24.120 INFO:tasks.workunit.client.0.smithi172.stderr:3 2024-05-08T16:14:24.120 INFO:tasks.workunit.client.0.smithi172.stderr:1 2024-05-08T16:14:24.120 INFO:tasks.workunit.client.0.smithi172.stderr:5 2024-05-08T16:14:24.120 INFO:tasks.workunit.client.0.smithi172.stderr:2' 2024-05-08T16:14:24.120 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 4 10 0 7 6 8 3 1 5 2 2024-05-08T16:14:24.120 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:96: rados_put_get: rados --pool pool-isa get SOMETHING td/test-erasure-code/COPY 2024-05-08T16:14:24.215 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:97: rados_put_get: diff td/test-erasure-code/ORIGINAL td/test-erasure-code/COPY 2024-05-08T16:14:24.216 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:98: rados_put_get: ceph osd in 9 2024-05-08T16:14:24.807 INFO:tasks.workunit.client.0.smithi172.stderr:osd.9 is already in. 2024-05-08T16:14:24.825 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:100: rados_put_get: rm td/test-erasure-code/ORIGINAL 2024-05-08T16:14:24.827 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:208: TEST_rados_put_get_isa: delete_pool pool-isa 2024-05-08T16:14:24.827 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=pool-isa 2024-05-08T16:14:24.827 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete pool-isa pool-isa --yes-i-really-really-mean-it 2024-05-08T16:14:25.924 INFO:tasks.workunit.client.0.smithi172.stderr:pool 'pool-isa' does not exist 2024-05-08T16:14:25.943 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:47: run: for func in $funcs 2024-05-08T16:14:25.943 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:48: run: TEST_rados_put_get_jerasure td/test-erasure-code 2024-05-08T16:14:25.944 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:212: TEST_rados_put_get_jerasure: local dir=td/test-erasure-code 2024-05-08T16:14:25.944 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:214: TEST_rados_put_get_jerasure: rados_put_get td/test-erasure-code ecpool 2024-05-08T16:14:25.944 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:66: rados_put_get: local dir=td/test-erasure-code 2024-05-08T16:14:25.944 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:67: rados_put_get: local poolname=ecpool 2024-05-08T16:14:25.944 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:68: rados_put_get: local objname=SOMETHING 2024-05-08T16:14:25.944 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2024-05-08T16:14:25.944 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 AAA 2024-05-08T16:14:25.944 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2024-05-08T16:14:25.944 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 BBB 2024-05-08T16:14:25.944 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2024-05-08T16:14:25.945 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 CCCC 2024-05-08T16:14:25.945 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2024-05-08T16:14:25.945 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 DDDD 2024-05-08T16:14:25.945 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:78: rados_put_get: rados --pool ecpool put SOMETHING td/test-erasure-code/ORIGINAL 2024-05-08T16:14:27.396 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:79: rados_put_get: rados --pool ecpool get SOMETHING td/test-erasure-code/COPY 2024-05-08T16:14:27.615 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:80: rados_put_get: diff td/test-erasure-code/ORIGINAL td/test-erasure-code/COPY 2024-05-08T16:14:27.616 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:81: rados_put_get: rm td/test-erasure-code/COPY 2024-05-08T16:14:27.645 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:88: rados_put_get: get_osds ecpool SOMETHING 2024-05-08T16:14:27.679 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=ecpool 2024-05-08T16:14:27.679 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=SOMETHING 2024-05-08T16:14:27.679 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map ecpool SOMETHING 2024-05-08T16:14:27.680 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-05-08T16:14:28.154 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=6 2024-05-08T16:14:28.154 INFO:tasks.workunit.client.0.smithi172.stderr:0 2024-05-08T16:14:28.154 INFO:tasks.workunit.client.0.smithi172.stderr:7 2024-05-08T16:14:28.154 INFO:tasks.workunit.client.0.smithi172.stderr:9' 2024-05-08T16:14:28.154 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 6 0 7 9 2024-05-08T16:14:28.154 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:88: rados_put_get: initial_osds=('6' '0' '7' '9') 2024-05-08T16:14:28.154 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:88: rados_put_get: local -a initial_osds 2024-05-08T16:14:28.154 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:89: rados_put_get: local last=3 2024-05-08T16:14:28.154 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:90: rados_put_get: ceph osd out 9 2024-05-08T16:14:28.797 INFO:tasks.workunit.client.0.smithi172.stderr:osd.9 is already out. 2024-05-08T16:14:28.816 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:93: rados_put_get: sleep 5 2024-05-08T16:14:33.818 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:94: rados_put_get: get_osds ecpool SOMETHING 2024-05-08T16:14:33.818 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=ecpool 2024-05-08T16:14:33.819 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=SOMETHING 2024-05-08T16:14:33.819 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:94: rados_put_get: grep '\<9\>' 2024-05-08T16:14:33.819 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map ecpool SOMETHING 2024-05-08T16:14:33.819 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-05-08T16:14:34.232 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=6 2024-05-08T16:14:34.232 INFO:tasks.workunit.client.0.smithi172.stderr:0 2024-05-08T16:14:34.232 INFO:tasks.workunit.client.0.smithi172.stderr:7 2024-05-08T16:14:34.232 INFO:tasks.workunit.client.0.smithi172.stderr:5' 2024-05-08T16:14:34.232 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 6 0 7 5 2024-05-08T16:14:34.232 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:96: rados_put_get: rados --pool ecpool get SOMETHING td/test-erasure-code/COPY 2024-05-08T16:14:34.325 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:97: rados_put_get: diff td/test-erasure-code/ORIGINAL td/test-erasure-code/COPY 2024-05-08T16:14:34.326 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:98: rados_put_get: ceph osd in 9 2024-05-08T16:14:34.840 INFO:tasks.workunit.client.0.smithi172.stderr:osd.9 is already in. 2024-05-08T16:14:34.856 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:100: rados_put_get: rm td/test-erasure-code/ORIGINAL 2024-05-08T16:14:34.857 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:216: TEST_rados_put_get_jerasure: local poolname=pool-jerasure 2024-05-08T16:14:34.857 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:217: TEST_rados_put_get_jerasure: local profile=profile-jerasure 2024-05-08T16:14:34.857 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:219: TEST_rados_put_get_jerasure: ceph osd erasure-code-profile set profile-jerasure plugin=jerasure k=4 m=2 crush-failure-domain=osd 2024-05-08T16:14:37.004 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:223: TEST_rados_put_get_jerasure: create_pool pool-jerasure 12 12 erasure profile-jerasure 2024-05-08T16:14:37.004 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create pool-jerasure 12 12 erasure profile-jerasure 2024-05-08T16:14:38.738 INFO:tasks.workunit.client.0.smithi172.stderr:pool 'pool-jerasure' already exists 2024-05-08T16:14:38.759 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-05-08T16:14:39.762 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:226: TEST_rados_put_get_jerasure: rados_put_get td/test-erasure-code pool-jerasure 2024-05-08T16:14:39.762 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:66: rados_put_get: local dir=td/test-erasure-code 2024-05-08T16:14:39.763 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:67: rados_put_get: local poolname=pool-jerasure 2024-05-08T16:14:39.763 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:68: rados_put_get: local objname=SOMETHING 2024-05-08T16:14:39.763 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2024-05-08T16:14:39.763 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 AAA 2024-05-08T16:14:39.763 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2024-05-08T16:14:39.763 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 BBB 2024-05-08T16:14:39.763 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2024-05-08T16:14:39.763 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 CCCC 2024-05-08T16:14:39.763 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2024-05-08T16:14:39.763 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 DDDD 2024-05-08T16:14:39.763 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:78: rados_put_get: rados --pool pool-jerasure put SOMETHING td/test-erasure-code/ORIGINAL 2024-05-08T16:14:41.735 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:79: rados_put_get: rados --pool pool-jerasure get SOMETHING td/test-erasure-code/COPY 2024-05-08T16:14:42.256 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:80: rados_put_get: diff td/test-erasure-code/ORIGINAL td/test-erasure-code/COPY 2024-05-08T16:14:42.258 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:81: rados_put_get: rm td/test-erasure-code/COPY 2024-05-08T16:14:42.285 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:88: rados_put_get: get_osds pool-jerasure SOMETHING 2024-05-08T16:14:42.286 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2024-05-08T16:14:42.286 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=SOMETHING 2024-05-08T16:14:42.286 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure SOMETHING 2024-05-08T16:14:42.286 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-05-08T16:14:42.967 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=5 2024-05-08T16:14:42.967 INFO:tasks.workunit.client.0.smithi172.stderr:1 2024-05-08T16:14:42.967 INFO:tasks.workunit.client.0.smithi172.stderr:9 2024-05-08T16:14:42.967 INFO:tasks.workunit.client.0.smithi172.stderr:4 2024-05-08T16:14:42.967 INFO:tasks.workunit.client.0.smithi172.stderr:3 2024-05-08T16:14:42.967 INFO:tasks.workunit.client.0.smithi172.stderr:10' 2024-05-08T16:14:42.967 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 5 1 9 4 3 10 2024-05-08T16:14:42.967 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:88: rados_put_get: initial_osds=('5' '1' '9' '4' '3' '10') 2024-05-08T16:14:42.968 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:88: rados_put_get: local -a initial_osds 2024-05-08T16:14:42.968 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:89: rados_put_get: local last=5 2024-05-08T16:14:42.968 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:90: rados_put_get: ceph osd out 10 2024-05-08T16:14:43.771 INFO:tasks.workunit.client.0.smithi172.stderr:osd.10 is already out. 2024-05-08T16:14:43.793 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:93: rados_put_get: sleep 5 2024-05-08T16:14:48.795 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:94: rados_put_get: get_osds pool-jerasure SOMETHING 2024-05-08T16:14:48.795 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2024-05-08T16:14:48.795 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=SOMETHING 2024-05-08T16:14:48.795 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:94: rados_put_get: grep '\<10\>' 2024-05-08T16:14:48.796 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure SOMETHING 2024-05-08T16:14:48.796 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-05-08T16:14:49.221 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=5 2024-05-08T16:14:49.221 INFO:tasks.workunit.client.0.smithi172.stderr:1 2024-05-08T16:14:49.222 INFO:tasks.workunit.client.0.smithi172.stderr:9 2024-05-08T16:14:49.222 INFO:tasks.workunit.client.0.smithi172.stderr:4 2024-05-08T16:14:49.222 INFO:tasks.workunit.client.0.smithi172.stderr:3 2024-05-08T16:14:49.222 INFO:tasks.workunit.client.0.smithi172.stderr:6' 2024-05-08T16:14:49.222 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 5 1 9 4 3 6 2024-05-08T16:14:49.222 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:96: rados_put_get: rados --pool pool-jerasure get SOMETHING td/test-erasure-code/COPY 2024-05-08T16:14:49.334 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:97: rados_put_get: diff td/test-erasure-code/ORIGINAL td/test-erasure-code/COPY 2024-05-08T16:14:49.335 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:98: rados_put_get: ceph osd in 10 2024-05-08T16:14:49.945 INFO:tasks.workunit.client.0.smithi172.stderr:osd.10 is already in. 2024-05-08T16:14:49.963 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:100: rados_put_get: rm td/test-erasure-code/ORIGINAL 2024-05-08T16:14:49.964 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:227: TEST_rados_put_get_jerasure: rados_osds_out_in td/test-erasure-code pool-jerasure 2024-05-08T16:14:49.965 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:104: rados_osds_out_in: local dir=td/test-erasure-code 2024-05-08T16:14:49.965 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:105: rados_osds_out_in: local poolname=pool-jerasure 2024-05-08T16:14:49.965 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:106: rados_osds_out_in: local objname=SOMETHING 2024-05-08T16:14:49.965 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:109: rados_osds_out_in: for marker in FFFF GGGG HHHH IIII 2024-05-08T16:14:49.965 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:110: rados_osds_out_in: printf '%*s' 1024 FFFF 2024-05-08T16:14:49.965 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:109: rados_osds_out_in: for marker in FFFF GGGG HHHH IIII 2024-05-08T16:14:49.965 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:110: rados_osds_out_in: printf '%*s' 1024 GGGG 2024-05-08T16:14:49.965 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:109: rados_osds_out_in: for marker in FFFF GGGG HHHH IIII 2024-05-08T16:14:49.965 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:110: rados_osds_out_in: printf '%*s' 1024 HHHH 2024-05-08T16:14:49.965 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:109: rados_osds_out_in: for marker in FFFF GGGG HHHH IIII 2024-05-08T16:14:49.965 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:110: rados_osds_out_in: printf '%*s' 1024 IIII 2024-05-08T16:14:49.965 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:116: rados_osds_out_in: rados --pool pool-jerasure put SOMETHING td/test-erasure-code/ORIGINAL 2024-05-08T16:14:52.719 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:117: rados_osds_out_in: rados --pool pool-jerasure get SOMETHING td/test-erasure-code/COPY 2024-05-08T16:14:52.758 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:118: rados_osds_out_in: diff td/test-erasure-code/ORIGINAL td/test-erasure-code/COPY 2024-05-08T16:14:52.759 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:119: rados_osds_out_in: rm td/test-erasure-code/COPY 2024-05-08T16:14:52.772 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:127: rados_osds_out_in: wait_for_clean 2024-05-08T16:14:52.772 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-05-08T16:14:52.772 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-05-08T16:14:52.773 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-05-08T16:14:52.774 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-05-08T16:14:52.774 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-05-08T16:14:52.774 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-05-08T16:14:52.774 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-05-08T16:14:52.774 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-05-08T16:14:52.774 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-05-08T16:14:52.857 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-05-08T16:14:52.858 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-05-08T16:14:52.858 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-05-08T16:14:52.858 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-05-08T16:14:52.858 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-05-08T16:14:52.858 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-05-08T16:14:53.521 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-05-08T16:14:53.521 INFO:tasks.workunit.client.0.smithi172.stderr:1 2024-05-08T16:14:53.521 INFO:tasks.workunit.client.0.smithi172.stderr:2 2024-05-08T16:14:53.521 INFO:tasks.workunit.client.0.smithi172.stderr:3 2024-05-08T16:14:53.521 INFO:tasks.workunit.client.0.smithi172.stderr:4 2024-05-08T16:14:53.521 INFO:tasks.workunit.client.0.smithi172.stderr:5 2024-05-08T16:14:53.522 INFO:tasks.workunit.client.0.smithi172.stderr:6 2024-05-08T16:14:53.522 INFO:tasks.workunit.client.0.smithi172.stderr:7 2024-05-08T16:14:53.522 INFO:tasks.workunit.client.0.smithi172.stderr:8 2024-05-08T16:14:53.522 INFO:tasks.workunit.client.0.smithi172.stderr:9 2024-05-08T16:14:53.522 INFO:tasks.workunit.client.0.smithi172.stderr:10' 2024-05-08T16:14:53.522 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-05-08T16:14:53.522 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T16:14:53.523 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-05-08T16:14:53.901 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836607 2024-05-08T16:14:53.901 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836607 2024-05-08T16:14:53.901 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836607' 2024-05-08T16:14:53.901 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T16:14:53.902 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-05-08T16:14:54.061 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673077 2024-05-08T16:14:54.061 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673077 2024-05-08T16:14:54.061 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836607 1-42949673077' 2024-05-08T16:14:54.061 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T16:14:54.061 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-05-08T16:14:54.463 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509552 2024-05-08T16:14:54.463 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509552 2024-05-08T16:14:54.463 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836607 1-42949673077 2-64424509552' 2024-05-08T16:14:54.463 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T16:14:54.464 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-05-08T16:14:54.631 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=326417514530 2024-05-08T16:14:54.631 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 326417514530 2024-05-08T16:14:54.631 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836607 1-42949673077 2-64424509552 3-326417514530' 2024-05-08T16:14:54.631 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T16:14:54.631 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-05-08T16:14:54.788 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182505 2024-05-08T16:14:54.788 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182505 2024-05-08T16:14:54.788 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836607 1-42949673077 2-64424509552 3-326417514530 4-107374182505' 2024-05-08T16:14:54.788 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T16:14:54.788 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-05-08T16:14:54.941 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=356482285597 2024-05-08T16:14:54.941 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 356482285597 2024-05-08T16:14:54.942 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836607 1-42949673077 2-64424509552 3-326417514530 4-107374182505 5-356482285597' 2024-05-08T16:14:54.942 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T16:14:54.942 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-05-08T16:14:55.091 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=420906795030 2024-05-08T16:14:55.091 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 420906795030 2024-05-08T16:14:55.091 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836607 1-42949673077 2-64424509552 3-326417514530 4-107374182505 5-356482285597 6-420906795030' 2024-05-08T16:14:55.091 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T16:14:55.092 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-05-08T16:14:55.243 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691912 2024-05-08T16:14:55.243 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691912 2024-05-08T16:14:55.243 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836607 1-42949673077 2-64424509552 3-326417514530 4-107374182505 5-356482285597 6-420906795030 7-171798691912' 2024-05-08T16:14:55.243 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T16:14:55.243 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-05-08T16:14:55.395 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528387 2024-05-08T16:14:55.395 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528387 2024-05-08T16:14:55.395 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836607 1-42949673077 2-64424509552 3-326417514530 4-107374182505 5-356482285597 6-420906795030 7-171798691912 8-193273528387' 2024-05-08T16:14:55.395 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T16:14:55.395 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-05-08T16:14:55.548 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=450971566096 2024-05-08T16:14:55.548 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 450971566096 2024-05-08T16:14:55.548 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836607 1-42949673077 2-64424509552 3-326417514530 4-107374182505 5-356482285597 6-420906795030 7-171798691912 8-193273528387 9-450971566096' 2024-05-08T16:14:55.548 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T16:14:55.548 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.10 flush_pg_stats 2024-05-08T16:14:55.702 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=236223201332 2024-05-08T16:14:55.702 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 236223201332 2024-05-08T16:14:55.702 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836607 1-42949673077 2-64424509552 3-326417514530 4-107374182505 5-356482285597 6-420906795030 7-171798691912 8-193273528387 9-450971566096 10-236223201332' 2024-05-08T16:14:55.703 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-08T16:14:55.703 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836607 2024-05-08T16:14:55.703 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-08T16:14:55.704 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-05-08T16:14:55.704 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836607 2024-05-08T16:14:55.705 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-08T16:14:55.705 INFO:tasks.workunit.client.0.smithi172.stdout:waiting osd.0 seq 21474836607 2024-05-08T16:14:55.706 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836607 2024-05-08T16:14:55.706 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836607' 2024-05-08T16:14:55.706 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-05-08T16:14:56.116 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836607 -lt 21474836607 2024-05-08T16:14:56.116 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-08T16:14:56.117 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673077 2024-05-08T16:14:56.117 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-08T16:14:56.118 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-05-08T16:14:56.119 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673077 2024-05-08T16:14:56.119 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-08T16:14:56.120 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673077 2024-05-08T16:14:56.121 INFO:tasks.workunit.client.0.smithi172.stdout:waiting osd.1 seq 42949673077 2024-05-08T16:14:56.121 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673077' 2024-05-08T16:14:56.121 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-05-08T16:14:56.552 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673077 -lt 42949673077 2024-05-08T16:14:56.552 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-08T16:14:56.552 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509552 2024-05-08T16:14:56.552 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-08T16:14:56.554 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-05-08T16:14:56.555 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509552 2024-05-08T16:14:56.555 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-08T16:14:56.555 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509552 2024-05-08T16:14:56.556 INFO:tasks.workunit.client.0.smithi172.stdout:waiting osd.2 seq 64424509552 2024-05-08T16:14:56.556 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509552' 2024-05-08T16:14:56.556 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-05-08T16:14:56.983 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509552 -lt 64424509552 2024-05-08T16:14:56.983 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-08T16:14:56.983 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-08T16:14:56.984 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-326417514530 2024-05-08T16:14:56.985 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-05-08T16:14:56.986 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-326417514530 2024-05-08T16:14:56.986 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-08T16:14:56.987 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=326417514530 2024-05-08T16:14:56.987 INFO:tasks.workunit.client.0.smithi172.stdout:waiting osd.3 seq 326417514530 2024-05-08T16:14:56.988 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 326417514530' 2024-05-08T16:14:56.988 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-05-08T16:14:57.439 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 326417514531 -lt 326417514530 2024-05-08T16:14:57.439 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-08T16:14:57.440 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-08T16:14:57.440 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182505 2024-05-08T16:14:57.441 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-05-08T16:14:57.442 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182505 2024-05-08T16:14:57.442 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-08T16:14:57.443 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182505 2024-05-08T16:14:57.444 INFO:tasks.workunit.client.0.smithi172.stdout:waiting osd.4 seq 107374182505 2024-05-08T16:14:57.444 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182505' 2024-05-08T16:14:57.444 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-05-08T16:14:57.853 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182505 -lt 107374182505 2024-05-08T16:14:57.853 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-08T16:14:57.854 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-356482285597 2024-05-08T16:14:57.854 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-08T16:14:57.855 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-05-08T16:14:57.856 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-356482285597 2024-05-08T16:14:57.856 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-08T16:14:57.857 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=356482285597 2024-05-08T16:14:57.858 INFO:tasks.workunit.client.0.smithi172.stdout:waiting osd.5 seq 356482285597 2024-05-08T16:14:57.858 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 356482285597' 2024-05-08T16:14:57.858 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-05-08T16:14:58.284 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 356482285597 -lt 356482285597 2024-05-08T16:14:58.284 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-08T16:14:58.284 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-420906795030 2024-05-08T16:14:58.285 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-08T16:14:58.286 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-05-08T16:14:58.287 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-420906795030 2024-05-08T16:14:58.287 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-08T16:14:58.288 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=420906795030 2024-05-08T16:14:58.288 INFO:tasks.workunit.client.0.smithi172.stdout:waiting osd.6 seq 420906795030 2024-05-08T16:14:58.289 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 420906795030' 2024-05-08T16:14:58.289 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-05-08T16:14:58.731 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 420906795030 -lt 420906795030 2024-05-08T16:14:58.731 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-08T16:14:58.732 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691912 2024-05-08T16:14:58.732 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-08T16:14:58.734 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-05-08T16:14:58.735 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691912 2024-05-08T16:14:58.735 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-08T16:14:58.737 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691912 2024-05-08T16:14:58.737 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691912' 2024-05-08T16:14:58.737 INFO:tasks.workunit.client.0.smithi172.stdout:waiting osd.7 seq 171798691912 2024-05-08T16:14:58.737 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-05-08T16:14:59.118 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691913 -lt 171798691912 2024-05-08T16:14:59.118 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-08T16:14:59.119 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-193273528387 2024-05-08T16:14:59.119 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-08T16:14:59.120 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-05-08T16:14:59.121 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-08T16:14:59.121 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528387 2024-05-08T16:14:59.122 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528387 2024-05-08T16:14:59.122 INFO:tasks.workunit.client.0.smithi172.stdout:waiting osd.8 seq 193273528387 2024-05-08T16:14:59.123 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528387' 2024-05-08T16:14:59.123 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-05-08T16:14:59.560 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528388 -lt 193273528387 2024-05-08T16:14:59.560 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-08T16:14:59.562 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-450971566096 2024-05-08T16:14:59.562 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-08T16:14:59.563 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-05-08T16:14:59.564 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-08T16:14:59.565 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-450971566096 2024-05-08T16:14:59.565 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=450971566096 2024-05-08T16:14:59.566 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 450971566096' 2024-05-08T16:14:59.566 INFO:tasks.workunit.client.0.smithi172.stdout:waiting osd.9 seq 450971566096 2024-05-08T16:14:59.566 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-05-08T16:14:59.964 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 450971566097 -lt 450971566096 2024-05-08T16:14:59.965 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-08T16:14:59.965 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 10-236223201332 2024-05-08T16:14:59.965 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-08T16:14:59.966 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=10 2024-05-08T16:14:59.967 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 10-236223201332 2024-05-08T16:14:59.968 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-08T16:14:59.969 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=236223201332 2024-05-08T16:14:59.969 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.10 seq 236223201332' 2024-05-08T16:14:59.969 INFO:tasks.workunit.client.0.smithi172.stdout:waiting osd.10 seq 236223201332 2024-05-08T16:14:59.969 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 10 2024-05-08T16:15:00.372 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 236223201332 -lt 236223201332 2024-05-08T16:15:00.372 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-05-08T16:15:00.373 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-05-08T16:15:00.373 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-05-08T16:15:01.087 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 28 == 0 2024-05-08T16:15:01.087 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-05-08T16:15:01.088 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-05-08T16:15:01.089 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-05-08T16:15:01.089 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-05-08T16:15:01.089 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-05-08T16:15:01.089 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-05-08T16:15:01.089 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-05-08T16:15:01.505 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=28 2024-05-08T16:15:01.505 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-05-08T16:15:01.506 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-05-08T16:15:01.506 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-05-08T16:15:02.038 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 28 = 28 2024-05-08T16:15:02.038 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-05-08T16:15:02.039 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-05-08T16:15:02.039 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:128: rados_osds_out_in: get_osds pool-jerasure SOMETHING 2024-05-08T16:15:02.039 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2024-05-08T16:15:02.039 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=SOMETHING 2024-05-08T16:15:02.040 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure SOMETHING 2024-05-08T16:15:02.040 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-05-08T16:15:02.439 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=5 2024-05-08T16:15:02.439 INFO:tasks.workunit.client.0.smithi172.stderr:1 2024-05-08T16:15:02.439 INFO:tasks.workunit.client.0.smithi172.stderr:9 2024-05-08T16:15:02.439 INFO:tasks.workunit.client.0.smithi172.stderr:4 2024-05-08T16:15:02.439 INFO:tasks.workunit.client.0.smithi172.stderr:3 2024-05-08T16:15:02.439 INFO:tasks.workunit.client.0.smithi172.stderr:10' 2024-05-08T16:15:02.439 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 5 1 9 4 3 10 2024-05-08T16:15:02.439 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:128: rados_osds_out_in: local 'osds_list=5 1 9 4 3 10' 2024-05-08T16:15:02.439 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:129: rados_osds_out_in: osds=('5' '1' '9' '4' '3' '10') 2024-05-08T16:15:02.439 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:129: rados_osds_out_in: local -a osds 2024-05-08T16:15:02.439 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:130: rados_osds_out_in: for osd in 0 1 2024-05-08T16:15:02.440 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:131: rados_osds_out_in: ceph osd out 5 2024-05-08T16:15:02.964 INFO:tasks.workunit.client.0.smithi172.stderr:osd.5 is already out. 2024-05-08T16:15:02.983 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:130: rados_osds_out_in: for osd in 0 1 2024-05-08T16:15:02.983 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:131: rados_osds_out_in: ceph osd out 1 2024-05-08T16:15:04.812 INFO:tasks.workunit.client.0.smithi172.stderr:osd.1 is already out. 2024-05-08T16:15:04.832 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:133: rados_osds_out_in: wait_for_clean 2024-05-08T16:15:04.833 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-05-08T16:15:04.833 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-05-08T16:15:04.833 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-05-08T16:15:04.833 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-05-08T16:15:04.834 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-05-08T16:15:04.834 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-05-08T16:15:04.834 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-05-08T16:15:04.834 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-05-08T16:15:04.834 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-05-08T16:15:04.941 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-05-08T16:15:04.941 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-05-08T16:15:04.941 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-05-08T16:15:04.941 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-05-08T16:15:04.941 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-05-08T16:15:04.941 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-05-08T16:15:05.558 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-05-08T16:15:05.559 INFO:tasks.workunit.client.0.smithi172.stderr:1 2024-05-08T16:15:05.559 INFO:tasks.workunit.client.0.smithi172.stderr:2 2024-05-08T16:15:05.559 INFO:tasks.workunit.client.0.smithi172.stderr:3 2024-05-08T16:15:05.559 INFO:tasks.workunit.client.0.smithi172.stderr:4 2024-05-08T16:15:05.559 INFO:tasks.workunit.client.0.smithi172.stderr:5 2024-05-08T16:15:05.559 INFO:tasks.workunit.client.0.smithi172.stderr:6 2024-05-08T16:15:05.559 INFO:tasks.workunit.client.0.smithi172.stderr:7 2024-05-08T16:15:05.559 INFO:tasks.workunit.client.0.smithi172.stderr:8 2024-05-08T16:15:05.559 INFO:tasks.workunit.client.0.smithi172.stderr:9 2024-05-08T16:15:05.559 INFO:tasks.workunit.client.0.smithi172.stderr:10' 2024-05-08T16:15:05.559 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-05-08T16:15:05.559 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T16:15:05.560 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-05-08T16:15:06.013 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836611 2024-05-08T16:15:06.013 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836611 2024-05-08T16:15:06.013 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836611' 2024-05-08T16:15:06.013 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T16:15:06.013 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-05-08T16:15:06.171 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673081 2024-05-08T16:15:06.171 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673081 2024-05-08T16:15:06.171 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836611 1-42949673081' 2024-05-08T16:15:06.172 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T16:15:06.172 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-05-08T16:15:06.324 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509556 2024-05-08T16:15:06.325 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509556 2024-05-08T16:15:06.325 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836611 1-42949673081 2-64424509556' 2024-05-08T16:15:06.325 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T16:15:06.325 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-05-08T16:15:06.827 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=326417514535 2024-05-08T16:15:06.827 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 326417514535 2024-05-08T16:15:06.827 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836611 1-42949673081 2-64424509556 3-326417514535' 2024-05-08T16:15:06.827 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T16:15:06.827 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-05-08T16:15:07.398 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182509 2024-05-08T16:15:07.399 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182509 2024-05-08T16:15:07.399 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836611 1-42949673081 2-64424509556 3-326417514535 4-107374182509' 2024-05-08T16:15:07.399 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T16:15:07.399 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-05-08T16:15:07.557 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=356482285601 2024-05-08T16:15:07.558 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 356482285601 2024-05-08T16:15:07.558 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836611 1-42949673081 2-64424509556 3-326417514535 4-107374182509 5-356482285601' 2024-05-08T16:15:07.558 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T16:15:07.558 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-05-08T16:15:07.710 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=420906795034 2024-05-08T16:15:07.710 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 420906795034 2024-05-08T16:15:07.711 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836611 1-42949673081 2-64424509556 3-326417514535 4-107374182509 5-356482285601 6-420906795034' 2024-05-08T16:15:07.711 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T16:15:07.711 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-05-08T16:15:08.175 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691917 2024-05-08T16:15:08.175 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691917 2024-05-08T16:15:08.175 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836611 1-42949673081 2-64424509556 3-326417514535 4-107374182509 5-356482285601 6-420906795034 7-171798691917' 2024-05-08T16:15:08.175 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T16:15:08.176 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-05-08T16:15:08.331 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528392 2024-05-08T16:15:08.332 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528392 2024-05-08T16:15:08.332 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836611 1-42949673081 2-64424509556 3-326417514535 4-107374182509 5-356482285601 6-420906795034 7-171798691917 8-193273528392' 2024-05-08T16:15:08.332 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T16:15:08.332 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-05-08T16:15:08.676 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=450971566101 2024-05-08T16:15:08.676 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 450971566101 2024-05-08T16:15:08.676 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836611 1-42949673081 2-64424509556 3-326417514535 4-107374182509 5-356482285601 6-420906795034 7-171798691917 8-193273528392 9-450971566101' 2024-05-08T16:15:08.676 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T16:15:08.676 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.10 flush_pg_stats 2024-05-08T16:15:08.828 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=236223201337 2024-05-08T16:15:08.828 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 236223201337 2024-05-08T16:15:08.828 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836611 1-42949673081 2-64424509556 3-326417514535 4-107374182509 5-356482285601 6-420906795034 7-171798691917 8-193273528392 9-450971566101 10-236223201337' 2024-05-08T16:15:08.828 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-08T16:15:08.829 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836611 2024-05-08T16:15:08.829 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-08T16:15:08.829 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-05-08T16:15:08.830 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836611 2024-05-08T16:15:08.830 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-08T16:15:08.831 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836611 2024-05-08T16:15:08.832 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836611' 2024-05-08T16:15:08.832 INFO:tasks.workunit.client.0.smithi172.stdout:waiting osd.0 seq 21474836611 2024-05-08T16:15:08.832 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-05-08T16:15:09.439 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836611 -lt 21474836611 2024-05-08T16:15:09.439 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-08T16:15:09.440 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673081 2024-05-08T16:15:09.440 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-08T16:15:09.442 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-05-08T16:15:09.443 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673081 2024-05-08T16:15:09.443 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-08T16:15:09.444 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673081 2024-05-08T16:15:09.444 INFO:tasks.workunit.client.0.smithi172.stdout:waiting osd.1 seq 42949673081 2024-05-08T16:15:09.445 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673081' 2024-05-08T16:15:09.445 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-05-08T16:15:09.901 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673082 -lt 42949673081 2024-05-08T16:15:09.901 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-08T16:15:09.902 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509556 2024-05-08T16:15:09.902 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-08T16:15:09.904 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-05-08T16:15:09.905 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509556 2024-05-08T16:15:09.905 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-08T16:15:09.906 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509556 2024-05-08T16:15:09.906 INFO:tasks.workunit.client.0.smithi172.stdout:waiting osd.2 seq 64424509556 2024-05-08T16:15:09.906 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509556' 2024-05-08T16:15:09.907 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-05-08T16:15:10.313 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509556 -lt 64424509556 2024-05-08T16:15:10.313 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-08T16:15:10.314 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-326417514535 2024-05-08T16:15:10.314 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-08T16:15:10.315 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-05-08T16:15:10.316 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-326417514535 2024-05-08T16:15:10.317 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-08T16:15:10.318 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=326417514535 2024-05-08T16:15:10.318 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 326417514535' 2024-05-08T16:15:10.318 INFO:tasks.workunit.client.0.smithi172.stdout:waiting osd.3 seq 326417514535 2024-05-08T16:15:10.319 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-05-08T16:15:10.713 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 326417514535 -lt 326417514535 2024-05-08T16:15:10.714 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-08T16:15:10.715 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182509 2024-05-08T16:15:10.715 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-08T16:15:10.716 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-05-08T16:15:10.717 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182509 2024-05-08T16:15:10.717 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-08T16:15:10.718 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182509 2024-05-08T16:15:10.718 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182509' 2024-05-08T16:15:10.718 INFO:tasks.workunit.client.0.smithi172.stdout:waiting osd.4 seq 107374182509 2024-05-08T16:15:10.718 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-05-08T16:15:11.165 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182510 -lt 107374182509 2024-05-08T16:15:11.165 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-08T16:15:11.167 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-356482285601 2024-05-08T16:15:11.167 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-08T16:15:11.168 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-05-08T16:15:11.169 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-356482285601 2024-05-08T16:15:11.169 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-08T16:15:11.170 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=356482285601 2024-05-08T16:15:11.171 INFO:tasks.workunit.client.0.smithi172.stdout:waiting osd.5 seq 356482285601 2024-05-08T16:15:11.172 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 356482285601' 2024-05-08T16:15:11.172 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-05-08T16:15:11.597 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 356482285602 -lt 356482285601 2024-05-08T16:15:11.597 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-08T16:15:11.598 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-420906795034 2024-05-08T16:15:11.598 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-08T16:15:11.600 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-05-08T16:15:11.601 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-420906795034 2024-05-08T16:15:11.601 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-08T16:15:11.602 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=420906795034 2024-05-08T16:15:11.602 INFO:tasks.workunit.client.0.smithi172.stdout:waiting osd.6 seq 420906795034 2024-05-08T16:15:11.602 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 420906795034' 2024-05-08T16:15:11.603 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-05-08T16:15:12.049 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 420906795035 -lt 420906795034 2024-05-08T16:15:12.050 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-08T16:15:12.050 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691917 2024-05-08T16:15:12.051 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-08T16:15:12.053 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-05-08T16:15:12.053 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691917 2024-05-08T16:15:12.053 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-08T16:15:12.054 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691917 2024-05-08T16:15:12.054 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691917' 2024-05-08T16:15:12.055 INFO:tasks.workunit.client.0.smithi172.stdout:waiting osd.7 seq 171798691917 2024-05-08T16:15:12.055 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-05-08T16:15:12.475 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691917 -lt 171798691917 2024-05-08T16:15:12.475 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-08T16:15:12.476 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-193273528392 2024-05-08T16:15:12.476 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-08T16:15:12.478 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-05-08T16:15:12.479 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528392 2024-05-08T16:15:12.479 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-08T16:15:12.481 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528392 2024-05-08T16:15:12.481 INFO:tasks.workunit.client.0.smithi172.stdout:waiting osd.8 seq 193273528392 2024-05-08T16:15:12.481 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528392' 2024-05-08T16:15:12.481 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-05-08T16:15:12.875 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528393 -lt 193273528392 2024-05-08T16:15:12.876 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-08T16:15:12.876 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-08T16:15:12.876 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-450971566101 2024-05-08T16:15:12.877 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-05-08T16:15:12.879 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-450971566101 2024-05-08T16:15:12.879 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-08T16:15:12.880 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=450971566101 2024-05-08T16:15:12.880 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 450971566101' 2024-05-08T16:15:12.880 INFO:tasks.workunit.client.0.smithi172.stdout:waiting osd.9 seq 450971566101 2024-05-08T16:15:12.881 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-05-08T16:15:13.332 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 450971566101 -lt 450971566101 2024-05-08T16:15:13.332 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-08T16:15:13.333 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 10-236223201337 2024-05-08T16:15:13.333 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-08T16:15:13.334 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=10 2024-05-08T16:15:13.335 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 10-236223201337 2024-05-08T16:15:13.335 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-08T16:15:13.336 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=236223201337 2024-05-08T16:15:13.336 INFO:tasks.workunit.client.0.smithi172.stdout:waiting osd.10 seq 236223201337 2024-05-08T16:15:13.337 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.10 seq 236223201337' 2024-05-08T16:15:13.337 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 10 2024-05-08T16:15:13.735 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 236223201337 -lt 236223201337 2024-05-08T16:15:13.736 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-05-08T16:15:13.736 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-05-08T16:15:13.736 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-05-08T16:15:14.292 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 28 == 0 2024-05-08T16:15:14.293 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-05-08T16:15:14.293 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-05-08T16:15:14.293 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-05-08T16:15:14.293 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-05-08T16:15:14.293 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-05-08T16:15:14.294 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-05-08T16:15:14.294 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-05-08T16:15:14.686 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=28 2024-05-08T16:15:14.686 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-05-08T16:15:14.687 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-05-08T16:15:14.687 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-05-08T16:15:15.243 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 28 = 28 2024-05-08T16:15:15.243 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-05-08T16:15:15.243 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-05-08T16:15:15.243 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:137: rados_osds_out_in: for osd in 0 1 2024-05-08T16:15:15.243 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:138: rados_osds_out_in: get_osds pool-jerasure SOMETHING 2024-05-08T16:15:15.243 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:138: rados_osds_out_in: grep '\<5\>' 2024-05-08T16:15:15.243 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2024-05-08T16:15:15.243 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=SOMETHING 2024-05-08T16:15:15.244 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure SOMETHING 2024-05-08T16:15:15.244 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-05-08T16:15:15.642 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=2 2024-05-08T16:15:15.643 INFO:tasks.workunit.client.0.smithi172.stderr:7 2024-05-08T16:15:15.643 INFO:tasks.workunit.client.0.smithi172.stderr:9 2024-05-08T16:15:15.643 INFO:tasks.workunit.client.0.smithi172.stderr:4 2024-05-08T16:15:15.643 INFO:tasks.workunit.client.0.smithi172.stderr:3 2024-05-08T16:15:15.643 INFO:tasks.workunit.client.0.smithi172.stderr:10' 2024-05-08T16:15:15.643 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 2 7 9 4 3 10 2024-05-08T16:15:15.643 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:137: rados_osds_out_in: for osd in 0 1 2024-05-08T16:15:15.644 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:138: rados_osds_out_in: get_osds pool-jerasure SOMETHING 2024-05-08T16:15:15.644 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:138: rados_osds_out_in: grep '\<1\>' 2024-05-08T16:15:15.644 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2024-05-08T16:15:15.644 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=SOMETHING 2024-05-08T16:15:15.644 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure SOMETHING 2024-05-08T16:15:15.644 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-05-08T16:15:16.085 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=2 2024-05-08T16:15:16.085 INFO:tasks.workunit.client.0.smithi172.stderr:7 2024-05-08T16:15:16.085 INFO:tasks.workunit.client.0.smithi172.stderr:9 2024-05-08T16:15:16.085 INFO:tasks.workunit.client.0.smithi172.stderr:4 2024-05-08T16:15:16.085 INFO:tasks.workunit.client.0.smithi172.stderr:3 2024-05-08T16:15:16.086 INFO:tasks.workunit.client.0.smithi172.stderr:10' 2024-05-08T16:15:16.086 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 2 7 9 4 3 10 2024-05-08T16:15:16.086 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:140: rados_osds_out_in: rados --pool pool-jerasure get SOMETHING td/test-erasure-code/COPY 2024-05-08T16:15:16.146 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:141: rados_osds_out_in: diff td/test-erasure-code/ORIGINAL td/test-erasure-code/COPY 2024-05-08T16:15:16.147 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:147: rados_osds_out_in: for osd in 0 1 2024-05-08T16:15:16.147 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:148: rados_osds_out_in: ceph osd in 5 2024-05-08T16:15:16.840 INFO:tasks.workunit.client.0.smithi172.stderr:osd.5 is already in. 2024-05-08T16:15:16.857 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:147: rados_osds_out_in: for osd in 0 1 2024-05-08T16:15:16.857 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:148: rados_osds_out_in: ceph osd in 1 2024-05-08T16:15:18.494 INFO:tasks.workunit.client.0.smithi172.stderr:osd.1 is already in. 2024-05-08T16:15:18.515 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:150: rados_osds_out_in: wait_for_clean 2024-05-08T16:15:18.516 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-05-08T16:15:18.516 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-05-08T16:15:18.516 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-05-08T16:15:18.516 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-05-08T16:15:18.517 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-05-08T16:15:18.517 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-05-08T16:15:18.517 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-05-08T16:15:18.518 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-05-08T16:15:18.518 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-05-08T16:15:18.623 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-05-08T16:15:18.623 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-05-08T16:15:18.624 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-05-08T16:15:18.624 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-05-08T16:15:18.624 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-05-08T16:15:18.624 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-05-08T16:15:19.738 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-05-08T16:15:19.738 INFO:tasks.workunit.client.0.smithi172.stderr:1 2024-05-08T16:15:19.738 INFO:tasks.workunit.client.0.smithi172.stderr:2 2024-05-08T16:15:19.739 INFO:tasks.workunit.client.0.smithi172.stderr:3 2024-05-08T16:15:19.739 INFO:tasks.workunit.client.0.smithi172.stderr:4 2024-05-08T16:15:19.739 INFO:tasks.workunit.client.0.smithi172.stderr:5 2024-05-08T16:15:19.739 INFO:tasks.workunit.client.0.smithi172.stderr:6 2024-05-08T16:15:19.739 INFO:tasks.workunit.client.0.smithi172.stderr:7 2024-05-08T16:15:19.739 INFO:tasks.workunit.client.0.smithi172.stderr:8 2024-05-08T16:15:19.739 INFO:tasks.workunit.client.0.smithi172.stderr:9 2024-05-08T16:15:19.739 INFO:tasks.workunit.client.0.smithi172.stderr:10' 2024-05-08T16:15:19.739 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-05-08T16:15:19.739 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T16:15:19.739 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-05-08T16:15:20.249 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=21474836616 2024-05-08T16:15:20.250 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 21474836616 2024-05-08T16:15:20.250 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836616' 2024-05-08T16:15:20.250 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T16:15:20.250 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-05-08T16:15:20.655 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=42949673086 2024-05-08T16:15:20.655 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 42949673086 2024-05-08T16:15:20.655 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836616 1-42949673086' 2024-05-08T16:15:20.655 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T16:15:20.655 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-05-08T16:15:21.224 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=64424509561 2024-05-08T16:15:21.224 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 64424509561 2024-05-08T16:15:21.224 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836616 1-42949673086 2-64424509561' 2024-05-08T16:15:21.224 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T16:15:21.225 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-05-08T16:15:21.385 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=326417514540 2024-05-08T16:15:21.385 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 326417514540 2024-05-08T16:15:21.385 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836616 1-42949673086 2-64424509561 3-326417514540' 2024-05-08T16:15:21.385 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T16:15:21.386 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-05-08T16:15:21.625 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=107374182514 2024-05-08T16:15:21.626 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 107374182514 2024-05-08T16:15:21.626 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836616 1-42949673086 2-64424509561 3-326417514540 4-107374182514' 2024-05-08T16:15:21.626 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T16:15:21.627 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-05-08T16:15:21.787 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=356482285606 2024-05-08T16:15:21.787 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 356482285606 2024-05-08T16:15:21.787 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836616 1-42949673086 2-64424509561 3-326417514540 4-107374182514 5-356482285606' 2024-05-08T16:15:21.787 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T16:15:21.788 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-05-08T16:15:22.191 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=420906795039 2024-05-08T16:15:22.191 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 420906795039 2024-05-08T16:15:22.191 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836616 1-42949673086 2-64424509561 3-326417514540 4-107374182514 5-356482285606 6-420906795039' 2024-05-08T16:15:22.191 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T16:15:22.191 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-05-08T16:15:22.378 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=171798691922 2024-05-08T16:15:22.378 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 171798691922 2024-05-08T16:15:22.378 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836616 1-42949673086 2-64424509561 3-326417514540 4-107374182514 5-356482285606 6-420906795039 7-171798691922' 2024-05-08T16:15:22.378 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T16:15:22.378 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-05-08T16:15:22.926 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=193273528397 2024-05-08T16:15:22.926 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 193273528397 2024-05-08T16:15:22.926 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836616 1-42949673086 2-64424509561 3-326417514540 4-107374182514 5-356482285606 6-420906795039 7-171798691922 8-193273528397' 2024-05-08T16:15:22.926 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T16:15:22.927 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2024-05-08T16:15:23.108 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=450971566106 2024-05-08T16:15:23.108 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 450971566106 2024-05-08T16:15:23.108 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836616 1-42949673086 2-64424509561 3-326417514540 4-107374182514 5-356482285606 6-420906795039 7-171798691922 8-193273528397 9-450971566106' 2024-05-08T16:15:23.108 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T16:15:23.108 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.10 flush_pg_stats 2024-05-08T16:15:23.275 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=236223201341 2024-05-08T16:15:23.275 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 236223201341 2024-05-08T16:15:23.275 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-21474836616 1-42949673086 2-64424509561 3-326417514540 4-107374182514 5-356482285606 6-420906795039 7-171798691922 8-193273528397 9-450971566106 10-236223201341' 2024-05-08T16:15:23.275 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-08T16:15:23.276 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-21474836616 2024-05-08T16:15:23.276 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-08T16:15:23.276 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-05-08T16:15:23.277 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-21474836616 2024-05-08T16:15:23.277 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-08T16:15:23.279 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=21474836616 2024-05-08T16:15:23.279 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 21474836616' 2024-05-08T16:15:23.279 INFO:tasks.workunit.client.0.smithi172.stdout:waiting osd.0 seq 21474836616 2024-05-08T16:15:23.279 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-05-08T16:15:23.681 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 21474836616 -lt 21474836616 2024-05-08T16:15:23.681 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-08T16:15:23.683 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-42949673086 2024-05-08T16:15:23.683 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-08T16:15:23.684 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-05-08T16:15:23.685 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-42949673086 2024-05-08T16:15:23.685 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-08T16:15:23.686 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=42949673086 2024-05-08T16:15:23.686 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 42949673086' 2024-05-08T16:15:23.687 INFO:tasks.workunit.client.0.smithi172.stdout:waiting osd.1 seq 42949673086 2024-05-08T16:15:23.687 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-05-08T16:15:24.110 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 42949673087 -lt 42949673086 2024-05-08T16:15:24.110 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-08T16:15:24.111 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-64424509561 2024-05-08T16:15:24.111 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-08T16:15:24.112 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-05-08T16:15:24.113 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-64424509561 2024-05-08T16:15:24.114 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-08T16:15:24.115 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=64424509561 2024-05-08T16:15:24.115 INFO:tasks.workunit.client.0.smithi172.stdout:waiting osd.2 seq 64424509561 2024-05-08T16:15:24.115 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 64424509561' 2024-05-08T16:15:24.115 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-05-08T16:15:24.582 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 64424509561 -lt 64424509561 2024-05-08T16:15:24.582 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-08T16:15:24.583 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-326417514540 2024-05-08T16:15:24.583 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-08T16:15:24.585 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-05-08T16:15:24.585 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-326417514540 2024-05-08T16:15:24.586 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-08T16:15:24.587 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=326417514540 2024-05-08T16:15:24.587 INFO:tasks.workunit.client.0.smithi172.stdout:waiting osd.3 seq 326417514540 2024-05-08T16:15:24.587 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 326417514540' 2024-05-08T16:15:24.588 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-05-08T16:15:24.990 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 326417514540 -lt 326417514540 2024-05-08T16:15:24.990 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-08T16:15:24.991 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-107374182514 2024-05-08T16:15:24.991 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-08T16:15:24.993 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-05-08T16:15:24.994 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-107374182514 2024-05-08T16:15:24.994 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-08T16:15:24.995 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=107374182514 2024-05-08T16:15:24.995 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 107374182514' 2024-05-08T16:15:24.995 INFO:tasks.workunit.client.0.smithi172.stdout:waiting osd.4 seq 107374182514 2024-05-08T16:15:24.996 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-05-08T16:15:25.451 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 107374182515 -lt 107374182514 2024-05-08T16:15:25.451 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-08T16:15:25.452 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-356482285606 2024-05-08T16:15:25.452 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-08T16:15:25.453 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-05-08T16:15:25.454 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-356482285606 2024-05-08T16:15:25.454 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-08T16:15:25.456 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=356482285606 2024-05-08T16:15:25.456 INFO:tasks.workunit.client.0.smithi172.stdout:waiting osd.5 seq 356482285606 2024-05-08T16:15:25.457 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 356482285606' 2024-05-08T16:15:25.457 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-05-08T16:15:25.872 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 356482285607 -lt 356482285606 2024-05-08T16:15:25.872 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-08T16:15:25.873 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-08T16:15:25.873 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-420906795039 2024-05-08T16:15:25.874 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-05-08T16:15:25.875 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-420906795039 2024-05-08T16:15:25.875 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-08T16:15:25.877 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=420906795039 2024-05-08T16:15:25.877 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 420906795039' 2024-05-08T16:15:25.877 INFO:tasks.workunit.client.0.smithi172.stdout:waiting osd.6 seq 420906795039 2024-05-08T16:15:25.877 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-05-08T16:15:26.313 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 420906795040 -lt 420906795039 2024-05-08T16:15:26.314 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-08T16:15:26.315 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-171798691922 2024-05-08T16:15:26.315 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-08T16:15:26.316 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-05-08T16:15:26.318 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-171798691922 2024-05-08T16:15:26.318 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-08T16:15:26.318 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=171798691922 2024-05-08T16:15:26.318 INFO:tasks.workunit.client.0.smithi172.stdout:waiting osd.7 seq 171798691922 2024-05-08T16:15:26.319 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 171798691922' 2024-05-08T16:15:26.319 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-05-08T16:15:26.729 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 171798691923 -lt 171798691922 2024-05-08T16:15:26.729 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-08T16:15:26.730 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-193273528397 2024-05-08T16:15:26.731 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-08T16:15:26.733 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-05-08T16:15:26.734 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-193273528397 2024-05-08T16:15:26.734 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-08T16:15:26.735 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=193273528397 2024-05-08T16:15:26.736 INFO:tasks.workunit.client.0.smithi172.stdout:waiting osd.8 seq 193273528397 2024-05-08T16:15:26.736 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 193273528397' 2024-05-08T16:15:26.736 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-05-08T16:15:27.124 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 193273528398 -lt 193273528397 2024-05-08T16:15:27.124 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-08T16:15:27.124 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 9-450971566106 2024-05-08T16:15:27.125 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-08T16:15:27.127 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=9 2024-05-08T16:15:27.128 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 9-450971566106 2024-05-08T16:15:27.128 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-08T16:15:27.129 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=450971566106 2024-05-08T16:15:27.129 INFO:tasks.workunit.client.0.smithi172.stdout:waiting osd.9 seq 450971566106 2024-05-08T16:15:27.130 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.9 seq 450971566106' 2024-05-08T16:15:27.130 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 9 2024-05-08T16:15:27.555 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 450971566106 -lt 450971566106 2024-05-08T16:15:27.555 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-08T16:15:27.556 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 10-236223201341 2024-05-08T16:15:27.556 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-08T16:15:27.557 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=10 2024-05-08T16:15:27.558 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 10-236223201341 2024-05-08T16:15:27.558 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-08T16:15:27.559 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=236223201341 2024-05-08T16:15:27.559 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.10 seq 236223201341' 2024-05-08T16:15:27.559 INFO:tasks.workunit.client.0.smithi172.stdout:waiting osd.10 seq 236223201341 2024-05-08T16:15:27.559 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 10 2024-05-08T16:15:28.003 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 236223201342 -lt 236223201341 2024-05-08T16:15:28.003 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-05-08T16:15:28.006 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-05-08T16:15:28.006 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-05-08T16:15:28.625 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 28 == 0 2024-05-08T16:15:28.625 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-05-08T16:15:28.626 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-05-08T16:15:28.626 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-05-08T16:15:28.626 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-05-08T16:15:28.626 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-05-08T16:15:28.626 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-05-08T16:15:28.626 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-05-08T16:15:29.086 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=28 2024-05-08T16:15:29.086 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-05-08T16:15:29.087 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-05-08T16:15:29.087 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-05-08T16:15:29.647 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 28 = 28 2024-05-08T16:15:29.647 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-05-08T16:15:29.647 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-05-08T16:15:29.647 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:151: rados_osds_out_in: get_osds pool-jerasure SOMETHING 2024-05-08T16:15:29.648 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2024-05-08T16:15:29.648 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=SOMETHING 2024-05-08T16:15:29.648 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-05-08T16:15:29.648 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure SOMETHING 2024-05-08T16:15:30.050 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=5 2024-05-08T16:15:30.050 INFO:tasks.workunit.client.0.smithi172.stderr:1 2024-05-08T16:15:30.050 INFO:tasks.workunit.client.0.smithi172.stderr:9 2024-05-08T16:15:30.050 INFO:tasks.workunit.client.0.smithi172.stderr:4 2024-05-08T16:15:30.050 INFO:tasks.workunit.client.0.smithi172.stderr:3 2024-05-08T16:15:30.050 INFO:tasks.workunit.client.0.smithi172.stderr:10' 2024-05-08T16:15:30.051 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 5 1 9 4 3 10 2024-05-08T16:15:30.051 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:151: rados_osds_out_in: test '5 1 9 4 3 10' = '5 1 9 4 3 10' 2024-05-08T16:15:30.051 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:152: rados_osds_out_in: rm td/test-erasure-code/ORIGINAL 2024-05-08T16:15:30.051 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:229: TEST_rados_put_get_jerasure: delete_pool pool-jerasure 2024-05-08T16:15:30.051 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=pool-jerasure 2024-05-08T16:15:30.051 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete pool-jerasure pool-jerasure --yes-i-really-really-mean-it 2024-05-08T16:15:30.555 INFO:tasks.workunit.client.0.smithi172.stderr:pool 'pool-jerasure' does not exist 2024-05-08T16:15:30.571 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:230: TEST_rados_put_get_jerasure: ceph osd erasure-code-profile rm profile-jerasure 2024-05-08T16:15:32.517 INFO:tasks.workunit.client.0.smithi172.stderr:erasure-code-profile profile-jerasure does not exist 2024-05-08T16:15:32.540 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:47: run: for func in $funcs 2024-05-08T16:15:32.540 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:48: run: TEST_rados_put_get_lrc_advanced td/test-erasure-code 2024-05-08T16:15:32.540 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:156: TEST_rados_put_get_lrc_advanced: local dir=td/test-erasure-code 2024-05-08T16:15:32.541 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:157: TEST_rados_put_get_lrc_advanced: local poolname=pool-lrc-a 2024-05-08T16:15:32.541 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:158: TEST_rados_put_get_lrc_advanced: local profile=profile-lrc-a 2024-05-08T16:15:32.541 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:160: TEST_rados_put_get_lrc_advanced: ceph osd erasure-code-profile set profile-lrc-a plugin=lrc mapping=DD_ 'crush-steps=[ [ "chooseleaf", "osd", 0 ] ]' 'layers=[ [ "DDc", "" ] ]' 2024-05-08T16:15:33.313 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:165: TEST_rados_put_get_lrc_advanced: create_pool pool-lrc-a 12 12 erasure profile-lrc-a 2024-05-08T16:15:33.313 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create pool-lrc-a 12 12 erasure profile-lrc-a 2024-05-08T16:15:34.730 INFO:tasks.workunit.client.0.smithi172.stderr:pool 'pool-lrc-a' already exists 2024-05-08T16:15:34.748 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-05-08T16:15:35.752 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:168: TEST_rados_put_get_lrc_advanced: rados_put_get td/test-erasure-code pool-lrc-a 2024-05-08T16:15:35.752 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:66: rados_put_get: local dir=td/test-erasure-code 2024-05-08T16:15:35.752 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:67: rados_put_get: local poolname=pool-lrc-a 2024-05-08T16:15:35.753 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:68: rados_put_get: local objname=SOMETHING 2024-05-08T16:15:35.753 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2024-05-08T16:15:35.753 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 AAA 2024-05-08T16:15:35.753 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2024-05-08T16:15:35.753 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 BBB 2024-05-08T16:15:35.753 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2024-05-08T16:15:35.753 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 CCCC 2024-05-08T16:15:35.754 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2024-05-08T16:15:35.754 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 DDDD 2024-05-08T16:15:35.754 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:78: rados_put_get: rados --pool pool-lrc-a put SOMETHING td/test-erasure-code/ORIGINAL 2024-05-08T16:15:37.680 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:79: rados_put_get: rados --pool pool-lrc-a get SOMETHING td/test-erasure-code/COPY 2024-05-08T16:15:37.929 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:80: rados_put_get: diff td/test-erasure-code/ORIGINAL td/test-erasure-code/COPY 2024-05-08T16:15:37.931 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:81: rados_put_get: rm td/test-erasure-code/COPY 2024-05-08T16:15:37.933 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:88: rados_put_get: get_osds pool-lrc-a SOMETHING 2024-05-08T16:15:37.933 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-lrc-a 2024-05-08T16:15:37.933 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=SOMETHING 2024-05-08T16:15:37.934 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-lrc-a SOMETHING 2024-05-08T16:15:37.934 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-05-08T16:15:38.728 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=5 2024-05-08T16:15:38.728 INFO:tasks.workunit.client.0.smithi172.stderr:9 2024-05-08T16:15:38.728 INFO:tasks.workunit.client.0.smithi172.stderr:3' 2024-05-08T16:15:38.729 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 5 9 3 2024-05-08T16:15:38.729 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:88: rados_put_get: initial_osds=('5' '9' '3') 2024-05-08T16:15:38.729 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:88: rados_put_get: local -a initial_osds 2024-05-08T16:15:38.729 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:89: rados_put_get: local last=2 2024-05-08T16:15:38.729 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:90: rados_put_get: ceph osd out 3 2024-05-08T16:15:39.605 INFO:tasks.workunit.client.0.smithi172.stderr:osd.3 is already out. 2024-05-08T16:15:39.626 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:93: rados_put_get: sleep 5 2024-05-08T16:15:44.629 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:94: rados_put_get: get_osds pool-lrc-a SOMETHING 2024-05-08T16:15:44.629 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-lrc-a 2024-05-08T16:15:44.629 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=SOMETHING 2024-05-08T16:15:44.629 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:94: rados_put_get: grep '\<3\>' 2024-05-08T16:15:44.630 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-lrc-a SOMETHING 2024-05-08T16:15:44.630 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-05-08T16:15:45.062 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=5 2024-05-08T16:15:45.062 INFO:tasks.workunit.client.0.smithi172.stderr:9 2024-05-08T16:15:45.062 INFO:tasks.workunit.client.0.smithi172.stderr:7' 2024-05-08T16:15:45.062 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 5 9 7 2024-05-08T16:15:45.062 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:96: rados_put_get: rados --pool pool-lrc-a get SOMETHING td/test-erasure-code/COPY 2024-05-08T16:15:45.134 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:97: rados_put_get: diff td/test-erasure-code/ORIGINAL td/test-erasure-code/COPY 2024-05-08T16:15:45.135 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:98: rados_put_get: ceph osd in 3 2024-05-08T16:15:45.696 INFO:tasks.workunit.client.0.smithi172.stderr:osd.3 is already in. 2024-05-08T16:15:45.712 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:100: rados_put_get: rm td/test-erasure-code/ORIGINAL 2024-05-08T16:15:45.713 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:170: TEST_rados_put_get_lrc_advanced: delete_pool pool-lrc-a 2024-05-08T16:15:45.713 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=pool-lrc-a 2024-05-08T16:15:45.713 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete pool-lrc-a pool-lrc-a --yes-i-really-really-mean-it 2024-05-08T16:15:47.266 INFO:tasks.workunit.client.0.smithi172.stderr:pool 'pool-lrc-a' does not exist 2024-05-08T16:15:47.286 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:171: TEST_rados_put_get_lrc_advanced: ceph osd erasure-code-profile rm profile-lrc-a 2024-05-08T16:15:48.999 INFO:tasks.workunit.client.0.smithi172.stderr:erasure-code-profile profile-lrc-a does not exist 2024-05-08T16:15:49.022 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:47: run: for func in $funcs 2024-05-08T16:15:49.022 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:48: run: TEST_rados_put_get_lrc_kml td/test-erasure-code 2024-05-08T16:15:49.022 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:175: TEST_rados_put_get_lrc_kml: local dir=td/test-erasure-code 2024-05-08T16:15:49.022 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:176: TEST_rados_put_get_lrc_kml: local poolname=pool-lrc 2024-05-08T16:15:49.022 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:177: TEST_rados_put_get_lrc_kml: local profile=profile-lrc 2024-05-08T16:15:49.023 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:179: TEST_rados_put_get_lrc_kml: ceph osd erasure-code-profile set profile-lrc plugin=lrc k=4 m=2 l=3 crush-failure-domain=osd 2024-05-08T16:15:49.844 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:183: TEST_rados_put_get_lrc_kml: create_pool pool-lrc 12 12 erasure profile-lrc 2024-05-08T16:15:49.844 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create pool-lrc 12 12 erasure profile-lrc 2024-05-08T16:15:50.915 INFO:tasks.workunit.client.0.smithi172.stderr:pool 'pool-lrc' already exists 2024-05-08T16:15:50.936 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-05-08T16:15:51.938 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:186: TEST_rados_put_get_lrc_kml: rados_put_get td/test-erasure-code pool-lrc 2024-05-08T16:15:51.938 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:66: rados_put_get: local dir=td/test-erasure-code 2024-05-08T16:15:51.939 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:67: rados_put_get: local poolname=pool-lrc 2024-05-08T16:15:51.939 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:68: rados_put_get: local objname=SOMETHING 2024-05-08T16:15:51.939 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2024-05-08T16:15:51.939 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 AAA 2024-05-08T16:15:51.939 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2024-05-08T16:15:51.939 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 BBB 2024-05-08T16:15:51.939 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2024-05-08T16:15:51.939 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 CCCC 2024-05-08T16:15:51.939 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2024-05-08T16:15:51.939 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 DDDD 2024-05-08T16:15:51.939 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:78: rados_put_get: rados --pool pool-lrc put SOMETHING td/test-erasure-code/ORIGINAL 2024-05-08T16:15:53.198 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:79: rados_put_get: rados --pool pool-lrc get SOMETHING td/test-erasure-code/COPY 2024-05-08T16:15:53.480 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:80: rados_put_get: diff td/test-erasure-code/ORIGINAL td/test-erasure-code/COPY 2024-05-08T16:15:53.482 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:81: rados_put_get: rm td/test-erasure-code/COPY 2024-05-08T16:15:53.498 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:88: rados_put_get: get_osds pool-lrc SOMETHING 2024-05-08T16:15:53.499 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-lrc 2024-05-08T16:15:53.499 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=SOMETHING 2024-05-08T16:15:53.500 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-lrc SOMETHING 2024-05-08T16:15:53.500 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-05-08T16:15:54.322 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=8 2024-05-08T16:15:54.323 INFO:tasks.workunit.client.0.smithi172.stderr:6 2024-05-08T16:15:54.323 INFO:tasks.workunit.client.0.smithi172.stderr:0 2024-05-08T16:15:54.323 INFO:tasks.workunit.client.0.smithi172.stderr:10 2024-05-08T16:15:54.323 INFO:tasks.workunit.client.0.smithi172.stderr:5 2024-05-08T16:15:54.323 INFO:tasks.workunit.client.0.smithi172.stderr:3 2024-05-08T16:15:54.323 INFO:tasks.workunit.client.0.smithi172.stderr:9 2024-05-08T16:15:54.323 INFO:tasks.workunit.client.0.smithi172.stderr:7' 2024-05-08T16:15:54.323 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 8 6 0 10 5 3 9 7 2024-05-08T16:15:54.323 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:88: rados_put_get: initial_osds=('8' '6' '0' '10' '5' '3' '9' '7') 2024-05-08T16:15:54.323 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:88: rados_put_get: local -a initial_osds 2024-05-08T16:15:54.323 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:89: rados_put_get: local last=7 2024-05-08T16:15:54.323 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:90: rados_put_get: ceph osd out 7 2024-05-08T16:15:55.083 INFO:tasks.workunit.client.0.smithi172.stderr:osd.7 is already out. 2024-05-08T16:15:55.100 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:93: rados_put_get: sleep 5 2024-05-08T16:16:00.102 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:94: rados_put_get: get_osds pool-lrc SOMETHING 2024-05-08T16:16:00.102 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-lrc 2024-05-08T16:16:00.102 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=SOMETHING 2024-05-08T16:16:00.103 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:94: rados_put_get: grep '\<7\>' 2024-05-08T16:16:00.103 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-lrc SOMETHING 2024-05-08T16:16:00.103 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-05-08T16:16:00.517 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=8 2024-05-08T16:16:00.518 INFO:tasks.workunit.client.0.smithi172.stderr:6 2024-05-08T16:16:00.518 INFO:tasks.workunit.client.0.smithi172.stderr:0 2024-05-08T16:16:00.518 INFO:tasks.workunit.client.0.smithi172.stderr:10 2024-05-08T16:16:00.518 INFO:tasks.workunit.client.0.smithi172.stderr:5 2024-05-08T16:16:00.518 INFO:tasks.workunit.client.0.smithi172.stderr:3 2024-05-08T16:16:00.518 INFO:tasks.workunit.client.0.smithi172.stderr:9 2024-05-08T16:16:00.518 INFO:tasks.workunit.client.0.smithi172.stderr:4' 2024-05-08T16:16:00.518 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 8 6 0 10 5 3 9 4 2024-05-08T16:16:00.518 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:96: rados_put_get: rados --pool pool-lrc get SOMETHING td/test-erasure-code/COPY 2024-05-08T16:16:00.643 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:97: rados_put_get: diff td/test-erasure-code/ORIGINAL td/test-erasure-code/COPY 2024-05-08T16:16:00.644 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:98: rados_put_get: ceph osd in 7 2024-05-08T16:16:01.457 INFO:tasks.workunit.client.0.smithi172.stderr:osd.7 is already in. 2024-05-08T16:16:01.482 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:100: rados_put_get: rm td/test-erasure-code/ORIGINAL 2024-05-08T16:16:01.483 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:188: TEST_rados_put_get_lrc_kml: delete_pool pool-lrc 2024-05-08T16:16:01.483 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=pool-lrc 2024-05-08T16:16:01.483 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete pool-lrc pool-lrc --yes-i-really-really-mean-it 2024-05-08T16:16:04.562 INFO:tasks.workunit.client.0.smithi172.stderr:pool 'pool-lrc' does not exist 2024-05-08T16:16:04.586 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:189: TEST_rados_put_get_lrc_kml: ceph osd erasure-code-profile rm profile-lrc 2024-05-08T16:16:06.294 INFO:tasks.workunit.client.0.smithi172.stderr:erasure-code-profile profile-lrc does not exist 2024-05-08T16:16:06.315 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:47: run: for func in $funcs 2024-05-08T16:16:06.315 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:48: run: TEST_rados_put_get_shec td/test-erasure-code 2024-05-08T16:16:06.315 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:234: TEST_rados_put_get_shec: local dir=td/test-erasure-code 2024-05-08T16:16:06.316 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:236: TEST_rados_put_get_shec: local poolname=pool-shec 2024-05-08T16:16:06.316 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:237: TEST_rados_put_get_shec: local profile=profile-shec 2024-05-08T16:16:06.316 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:239: TEST_rados_put_get_shec: ceph osd erasure-code-profile set profile-shec plugin=shec k=2 m=1 c=1 crush-failure-domain=osd 2024-05-08T16:16:07.283 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:243: TEST_rados_put_get_shec: create_pool pool-shec 12 12 erasure profile-shec 2024-05-08T16:16:07.283 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create pool-shec 12 12 erasure profile-shec 2024-05-08T16:16:08.429 INFO:tasks.workunit.client.0.smithi172.stderr:pool 'pool-shec' already exists 2024-05-08T16:16:08.445 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-05-08T16:16:09.451 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:246: TEST_rados_put_get_shec: rados_put_get td/test-erasure-code pool-shec 2024-05-08T16:16:09.451 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:66: rados_put_get: local dir=td/test-erasure-code 2024-05-08T16:16:09.451 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:67: rados_put_get: local poolname=pool-shec 2024-05-08T16:16:09.451 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:68: rados_put_get: local objname=SOMETHING 2024-05-08T16:16:09.451 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2024-05-08T16:16:09.452 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 AAA 2024-05-08T16:16:09.452 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2024-05-08T16:16:09.452 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 BBB 2024-05-08T16:16:09.452 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2024-05-08T16:16:09.452 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 CCCC 2024-05-08T16:16:09.452 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:71: rados_put_get: for marker in AAA BBB CCCC DDDD 2024-05-08T16:16:09.452 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:72: rados_put_get: printf '%*s' 1024 DDDD 2024-05-08T16:16:09.452 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:78: rados_put_get: rados --pool pool-shec put SOMETHING td/test-erasure-code/ORIGINAL 2024-05-08T16:16:11.801 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:79: rados_put_get: rados --pool pool-shec get SOMETHING td/test-erasure-code/COPY 2024-05-08T16:16:12.118 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:80: rados_put_get: diff td/test-erasure-code/ORIGINAL td/test-erasure-code/COPY 2024-05-08T16:16:12.118 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:81: rados_put_get: rm td/test-erasure-code/COPY 2024-05-08T16:16:12.126 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:88: rados_put_get: get_osds pool-shec SOMETHING 2024-05-08T16:16:12.126 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-shec 2024-05-08T16:16:12.126 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=SOMETHING 2024-05-08T16:16:12.127 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-shec SOMETHING 2024-05-08T16:16:12.127 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-05-08T16:16:12.856 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=7 2024-05-08T16:16:12.856 INFO:tasks.workunit.client.0.smithi172.stderr:9 2024-05-08T16:16:12.856 INFO:tasks.workunit.client.0.smithi172.stderr:1' 2024-05-08T16:16:12.857 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 7 9 1 2024-05-08T16:16:12.857 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:88: rados_put_get: initial_osds=('7' '9' '1') 2024-05-08T16:16:12.857 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:88: rados_put_get: local -a initial_osds 2024-05-08T16:16:12.857 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:89: rados_put_get: local last=2 2024-05-08T16:16:12.857 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:90: rados_put_get: ceph osd out 1 2024-05-08T16:16:13.592 INFO:tasks.workunit.client.0.smithi172.stderr:osd.1 is already out. 2024-05-08T16:16:13.609 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:93: rados_put_get: sleep 5 2024-05-08T16:16:18.612 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:94: rados_put_get: get_osds pool-shec SOMETHING 2024-05-08T16:16:18.612 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:94: rados_put_get: grep '\<1\>' 2024-05-08T16:16:18.613 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-shec 2024-05-08T16:16:18.613 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=SOMETHING 2024-05-08T16:16:18.614 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-shec SOMETHING 2024-05-08T16:16:18.614 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-05-08T16:16:19.137 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=7 2024-05-08T16:16:19.138 INFO:tasks.workunit.client.0.smithi172.stderr:9 2024-05-08T16:16:19.138 INFO:tasks.workunit.client.0.smithi172.stderr:6' 2024-05-08T16:16:19.138 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 7 9 6 2024-05-08T16:16:19.138 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:96: rados_put_get: rados --pool pool-shec get SOMETHING td/test-erasure-code/COPY 2024-05-08T16:16:19.223 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:97: rados_put_get: diff td/test-erasure-code/ORIGINAL td/test-erasure-code/COPY 2024-05-08T16:16:19.224 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:98: rados_put_get: ceph osd in 1 2024-05-08T16:16:19.804 INFO:tasks.workunit.client.0.smithi172.stderr:osd.1 is already in. 2024-05-08T16:16:19.821 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:100: rados_put_get: rm td/test-erasure-code/ORIGINAL 2024-05-08T16:16:19.822 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:248: TEST_rados_put_get_shec: delete_pool pool-shec 2024-05-08T16:16:19.822 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=pool-shec 2024-05-08T16:16:19.822 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete pool-shec pool-shec --yes-i-really-really-mean-it 2024-05-08T16:16:21.827 INFO:tasks.workunit.client.0.smithi172.stderr:pool 'pool-shec' does not exist 2024-05-08T16:16:21.847 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:249: TEST_rados_put_get_shec: ceph osd erasure-code-profile rm profile-shec 2024-05-08T16:16:23.033 INFO:tasks.workunit.client.0.smithi172.stderr:erasure-code-profile profile-shec does not exist 2024-05-08T16:16:23.052 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:51: run: delete_pool ecpool 2024-05-08T16:16:23.052 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=ecpool 2024-05-08T16:16:23.052 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete ecpool ecpool --yes-i-really-really-mean-it 2024-05-08T16:16:24.620 INFO:tasks.workunit.client.0.smithi172.stderr:pool 'ecpool' does not exist 2024-05-08T16:16:24.640 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-code.sh:52: run: teardown td/test-erasure-code 2024-05-08T16:16:24.640 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-code 2024-05-08T16:16:24.640 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-05-08T16:16:24.640 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-code KILL 2024-05-08T16:16:24.641 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-08T16:16:24.641 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-08T16:16:24.641 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-08T16:16:24.641 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-08T16:16:24.641 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-08T16:16:24.954 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-08T16:16:24.954 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-05-08T16:16:24.955 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-05-08T16:16:24.956 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-05-08T16:16:24.957 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-05-08T16:16:24.957 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-05-08T16:16:24.958 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-05-08T16:16:24.959 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-08T16:16:24.959 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-05-08T16:16:24.959 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-05-08T16:16:24.960 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-08T16:16:24.961 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-05-08T16:16:24.962 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-05-08T16:16:24.962 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-code 2024-05-08T16:16:25.081 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-05-08T16:16:25.081 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T16:16:25.081 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.48735 2024-05-08T16:16:25.081 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.48735 2024-05-08T16:16:25.083 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-05-08T16:16:25.083 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-05-08T16:16:25.083 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2314: main: code=0 2024-05-08T16:16:25.083 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2318: main: teardown td/test-erasure-code 0 2024-05-08T16:16:25.083 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-code 2024-05-08T16:16:25.083 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2024-05-08T16:16:25.083 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-code KILL 2024-05-08T16:16:25.083 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-08T16:16:25.083 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-08T16:16:25.083 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-08T16:16:25.083 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-08T16:16:25.083 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-08T16:16:25.086 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-08T16:16:25.086 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-05-08T16:16:25.086 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-05-08T16:16:25.087 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-05-08T16:16:25.088 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-05-08T16:16:25.088 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-05-08T16:16:25.088 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-05-08T16:16:25.089 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-08T16:16:25.089 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-05-08T16:16:25.090 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-05-08T16:16:25.090 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-08T16:16:25.090 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-05-08T16:16:25.092 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2024-05-08T16:16:25.092 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-code 2024-05-08T16:16:25.093 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-05-08T16:16:25.093 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T16:16:25.094 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.48735 2024-05-08T16:16:25.094 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.48735 2024-05-08T16:16:25.095 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-05-08T16:16:25.095 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-05-08T16:16:25.096 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2319: main: return 0 2024-05-08T16:16:25.097 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-05-08T16:16:25.097 DEBUG:teuthology.orchestra.run.smithi172:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-05-08T16:16:25.110 INFO:tasks.workunit:Running workunit erasure-code/test-erasure-eio.sh... 2024-05-08T16:16:25.111 DEBUG:teuthology.orchestra.run.smithi172:workunit test erasure-code/test-erasure-eio.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=95a4ad3c90f2f12f920e87725ad8d9002f948722 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh 2024-05-08T16:16:25.163 INFO:tasks.workunit.client.0.smithi172.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-05-08T16:16:25.169 INFO:tasks.workunit.client.0.smithi172.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-05-08T16:16:25.169 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2307: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-05-08T16:16:25.169 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2307: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-05-08T16:16:25.169 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2308: main: export PYTHONWARNINGS=ignore 2024-05-08T16:16:25.169 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2308: main: PYTHONWARNINGS=ignore 2024-05-08T16:16:25.170 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2309: main: export CEPH_CONF=/dev/null 2024-05-08T16:16:25.170 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2309: main: CEPH_CONF=/dev/null 2024-05-08T16:16:25.170 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2310: main: unset CEPH_ARGS 2024-05-08T16:16:25.170 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2312: main: local code 2024-05-08T16:16:25.170 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2313: main: run td/test-erasure-eio 2024-05-08T16:16:25.170 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:22: run: local dir=td/test-erasure-eio 2024-05-08T16:16:25.170 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:23: run: shift 2024-05-08T16:16:25.170 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:25: run: export CEPH_MON=127.0.0.1:7112 2024-05-08T16:16:25.170 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:25: run: CEPH_MON=127.0.0.1:7112 2024-05-08T16:16:25.170 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:26: run: export CEPH_ARGS 2024-05-08T16:16:25.170 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:27: run: uuidgen 2024-05-08T16:16:25.171 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:27: run: CEPH_ARGS+='--fsid=4457b0f1-028b-482b-8ee3-7ee05861e6cc --auth-supported=none ' 2024-05-08T16:16:25.171 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:28: run: CEPH_ARGS+='--mon-host=127.0.0.1:7112 ' 2024-05-08T16:16:25.171 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:29: run: CEPH_ARGS+='--osd_mclock_override_recovery_settings=true ' 2024-05-08T16:16:25.171 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:31: run: set 2024-05-08T16:16:25.171 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:31: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-05-08T16:16:25.173 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:31: run: local 'funcs=TEST_ec_backfill_unfound 2024-05-08T16:16:25.173 INFO:tasks.workunit.client.0.smithi172.stderr:TEST_ec_object_attr_read_error 2024-05-08T16:16:25.174 INFO:tasks.workunit.client.0.smithi172.stderr:TEST_ec_recovery_multiple_errors 2024-05-08T16:16:25.174 INFO:tasks.workunit.client.0.smithi172.stderr:TEST_ec_recovery_multiple_objects 2024-05-08T16:16:25.174 INFO:tasks.workunit.client.0.smithi172.stderr:TEST_ec_recovery_multiple_objects_eio 2024-05-08T16:16:25.174 INFO:tasks.workunit.client.0.smithi172.stderr:TEST_ec_recovery_unfound 2024-05-08T16:16:25.174 INFO:tasks.workunit.client.0.smithi172.stderr:TEST_ec_single_recovery_error 2024-05-08T16:16:25.174 INFO:tasks.workunit.client.0.smithi172.stderr:TEST_rados_get_bad_size_shard_0 2024-05-08T16:16:25.174 INFO:tasks.workunit.client.0.smithi172.stderr:TEST_rados_get_bad_size_shard_1 2024-05-08T16:16:25.174 INFO:tasks.workunit.client.0.smithi172.stderr:TEST_rados_get_subread_eio_shard_0 2024-05-08T16:16:25.174 INFO:tasks.workunit.client.0.smithi172.stderr:TEST_rados_get_subread_eio_shard_1 2024-05-08T16:16:25.174 INFO:tasks.workunit.client.0.smithi172.stderr:TEST_rados_get_subread_missing 2024-05-08T16:16:25.174 INFO:tasks.workunit.client.0.smithi172.stderr:TEST_rados_get_with_subreadall_eio_shard_0 2024-05-08T16:16:25.174 INFO:tasks.workunit.client.0.smithi172.stderr:TEST_rados_get_with_subreadall_eio_shard_1' 2024-05-08T16:16:25.174 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:32: run: for func in $funcs 2024-05-08T16:16:25.174 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:33: run: setup td/test-erasure-eio 2024-05-08T16:16:25.175 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/test-erasure-eio 2024-05-08T16:16:25.175 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/test-erasure-eio 2024-05-08T16:16:25.175 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-eio 2024-05-08T16:16:25.175 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-05-08T16:16:25.175 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-eio KILL 2024-05-08T16:16:25.175 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-08T16:16:25.175 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-08T16:16:25.175 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-08T16:16:25.175 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-08T16:16:25.175 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-08T16:16:25.176 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-08T16:16:25.176 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-05-08T16:16:25.176 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-05-08T16:16:25.176 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-05-08T16:16:25.178 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-05-08T16:16:25.178 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-05-08T16:16:25.178 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-05-08T16:16:25.179 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-08T16:16:25.179 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-05-08T16:16:25.179 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-05-08T16:16:25.179 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-08T16:16:25.180 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-05-08T16:16:25.181 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-05-08T16:16:25.181 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-eio 2024-05-08T16:16:25.182 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-05-08T16:16:25.183 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T16:16:25.183 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81110 2024-05-08T16:16:25.183 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.81110 2024-05-08T16:16:25.183 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-05-08T16:16:25.183 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-05-08T16:16:25.183 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/test-erasure-eio 2024-05-08T16:16:25.185 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-05-08T16:16:25.185 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T16:16:25.185 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81110 2024-05-08T16:16:25.185 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.81110 2024-05-08T16:16:25.187 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-05-08T16:16:25.187 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2024-05-08T16:16:25.187 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2024-05-08T16:16:25.188 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-05-08T16:16:25.188 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/test-erasure-eio 1' TERM HUP INT 2024-05-08T16:16:25.188 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:34: run: run_mon td/test-erasure-eio a 2024-05-08T16:16:25.188 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/test-erasure-eio 2024-05-08T16:16:25.188 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-05-08T16:16:25.188 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-05-08T16:16:25.188 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-05-08T16:16:25.188 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/test-erasure-eio/a 2024-05-08T16:16:25.188 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-eio/a --run-dir=td/test-erasure-eio 2024-05-08T16:16:25.588 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-05-08T16:16:25.588 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-08T16:16:25.588 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-08T16:16:25.589 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-08T16:16:25.589 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T16:16:25.589 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81110 2024-05-08T16:16:25.589 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.81110/$cluster-$name.asok' 2024-05-08T16:16:25.589 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-eio/a '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.81110/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-eio/log --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-05-08T16:16:25.811 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-05-08T16:16:25.811 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-05-08T16:16:25.811 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-05-08T16:16:25.811 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-05-08T16:16:25.812 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-05-08T16:16:25.812 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-05-08T16:16:25.812 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-05-08T16:16:25.812 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-05-08T16:16:25.812 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-05-08T16:16:25.812 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-08T16:16:25.812 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T16:16:25.812 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81110 2024-05-08T16:16:25.812 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.81110/ceph-mon.a.asok 2024-05-08T16:16:25.812 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-05-08T16:16:25.813 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.81110/ceph-mon.a.asok config get fsid 2024-05-08T16:16:25.929 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-05-08T16:16:25.929 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-05-08T16:16:25.929 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-05-08T16:16:25.929 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-05-08T16:16:25.929 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-05-08T16:16:25.931 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-05-08T16:16:25.931 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-05-08T16:16:25.931 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-05-08T16:16:25.931 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-08T16:16:25.931 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T16:16:25.931 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81110 2024-05-08T16:16:25.931 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.81110/ceph-mon.a.asok 2024-05-08T16:16:25.931 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-05-08T16:16:25.931 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.81110/ceph-mon.a.asok config get mon_host 2024-05-08T16:16:26.046 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:35: run: run_mgr td/test-erasure-eio x 2024-05-08T16:16:26.046 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/test-erasure-eio 2024-05-08T16:16:26.046 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-05-08T16:16:26.046 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-05-08T16:16:26.046 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-05-08T16:16:26.046 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/test-erasure-eio/x 2024-05-08T16:16:26.046 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-05-08T16:16:26.970 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-05-08T16:16:26.970 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-08T16:16:26.971 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-08T16:16:26.971 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-08T16:16:26.971 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T16:16:26.971 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81110 2024-05-08T16:16:26.971 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.81110/$cluster-$name.asok' 2024-05-08T16:16:26.971 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-05-08T16:16:26.972 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-eio/x '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.81110/$cluster-$name.asok' --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-05-08T16:16:27.003 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:36: run: create_pool rbd 4 2024-05-08T16:16:27.003 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-05-08T16:16:27.319 INFO:tasks.workunit.client.0.smithi172.stderr:pool 'rbd' already exists 2024-05-08T16:16:27.336 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-05-08T16:16:28.338 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: get_asok_path mon.a 2024-05-08T16:16:28.338 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-05-08T16:16:28.339 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-05-08T16:16:28.339 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-08T16:16:28.339 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T16:16:28.339 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81110 2024-05-08T16:16:28.339 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.81110/ceph-mon.a.asok 2024-05-08T16:16:28.340 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: CEPH_ARGS= 2024-05-08T16:16:28.340 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: ceph --admin-daemon /tmp/ceph-asok.81110/ceph-mon.a.asok log flush 2024-05-08T16:16:28.442 INFO:tasks.workunit.client.0.smithi172.stdout:{} 2024-05-08T16:16:28.453 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:40: run: grep 'load: jerasure.*lrc' td/test-erasure-eio/mon.a.log 2024-05-08T16:16:28.456 INFO:tasks.workunit.client.0.smithi172.stdout:2024-05-08T16:16:25.638+0000 7f925ac58c80 0 load: jerasure load: lrc 2024-05-08T16:16:28.457 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:41: run: TEST_ec_backfill_unfound td/test-erasure-eio 2024-05-08T16:16:28.457 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:511: TEST_ec_backfill_unfound: local dir=td/test-erasure-eio 2024-05-08T16:16:28.457 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:512: TEST_ec_backfill_unfound: local objname=myobject 2024-05-08T16:16:28.457 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:513: TEST_ec_backfill_unfound: local lastobj=300 2024-05-08T16:16:28.457 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:515: TEST_ec_backfill_unfound: local testobj=obj250 2024-05-08T16:16:28.457 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:517: TEST_ec_backfill_unfound: ORIG_ARGS='--fsid=4457b0f1-028b-482b-8ee3-7ee05861e6cc --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-05-08T16:16:28.458 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:518: TEST_ec_backfill_unfound: CEPH_ARGS+=' --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10' 2024-05-08T16:16:28.458 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:519: TEST_ec_backfill_unfound: setup_osds 5 2024-05-08T16:16:28.458 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:47: setup_osds: local count=5 2024-05-08T16:16:28.458 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:48: setup_osds: shift 2024-05-08T16:16:28.458 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: expr 5 - 1 2024-05-08T16:16:28.459 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: seq 0 4 2024-05-08T16:16:28.461 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-05-08T16:16:28.461 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 0 2024-05-08T16:16:28.461 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-05-08T16:16:28.461 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-05-08T16:16:28.461 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-05-08T16:16:28.461 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-05-08T16:16:28.461 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/0 2024-05-08T16:16:28.462 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=4457b0f1-028b-482b-8ee3-7ee05861e6cc --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10' 2024-05-08T16:16:28.462 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-08T16:16:28.462 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-05-08T16:16:28.462 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-08T16:16:28.462 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/0' 2024-05-08T16:16:28.462 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/0/journal' 2024-05-08T16:16:28.462 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-05-08T16:16:28.462 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-05-08T16:16:28.462 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-05-08T16:16:28.463 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-05-08T16:16:28.463 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-08T16:16:28.463 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-08T16:16:28.463 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-08T16:16:28.463 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T16:16:28.463 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81110 2024-05-08T16:16:28.464 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.81110/$cluster-$name.asok' 2024-05-08T16:16:28.466 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.81110/$cluster-$name.asok' 2024-05-08T16:16:28.466 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-05-08T16:16:28.466 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-05-08T16:16:28.466 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-05-08T16:16:28.466 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-05-08T16:16:28.466 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-05-08T16:16:28.466 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-08T16:16:28.466 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-08T16:16:28.466 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-08T16:16:28.466 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-08T16:16:28.466 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-05-08T16:16:28.467 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-05-08T16:16:28.467 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/0 2024-05-08T16:16:28.467 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-05-08T16:16:28.467 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=a6dc0bdd-651f-4bcf-a3e2-3dcb3c3a9048 2024-05-08T16:16:28.467 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 a6dc0bdd-651f-4bcf-a3e2-3dcb3c3a9048' 2024-05-08T16:16:28.467 INFO:tasks.workunit.client.0.smithi172.stdout:add osd0 a6dc0bdd-651f-4bcf-a3e2-3dcb3c3a9048 2024-05-08T16:16:28.467 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-05-08T16:16:28.486 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBcpTtmLKhvHRAAknH2nyW5pwFv1pgTDTBDQg== 2024-05-08T16:16:28.487 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBcpTtmLKhvHRAAknH2nyW5pwFv1pgTDTBDQg=="}' 2024-05-08T16:16:28.487 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new a6dc0bdd-651f-4bcf-a3e2-3dcb3c3a9048 -i td/test-erasure-eio/0/new.json 2024-05-08T16:16:28.786 INFO:tasks.workunit.client.0.smithi172.stdout:0 2024-05-08T16:16:28.802 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/0/new.json 2024-05-08T16:16:28.802 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=4457b0f1-028b-482b-8ee3-7ee05861e6cc --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81110/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBcpTtmLKhvHRAAknH2nyW5pwFv1pgTDTBDQg== --osd-uuid a6dc0bdd-651f-4bcf-a3e2-3dcb3c3a9048 2024-05-08T16:16:28.841 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:16:28.850+0000 7fb2288aa7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:16:28.855 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:16:28.862+0000 7fb2288aa7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:16:28.855 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:16:28.862+0000 7fb2288aa7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:16:28.856 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:16:28.862+0000 7fb2288aa7c0 -1 bluestore(td/test-erasure-eio/0/block) _read_bdev_label failed to open td/test-erasure-eio/0/block: (2) No such file or directory 2024-05-08T16:16:28.856 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:16:28.862+0000 7fb2288aa7c0 -1 bluestore(td/test-erasure-eio/0/block) _read_bdev_label failed to open td/test-erasure-eio/0/block: (2) No such file or directory 2024-05-08T16:16:28.856 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:16:28.862+0000 7fb2288aa7c0 -1 bluestore(td/test-erasure-eio/0/block) _read_bdev_label failed to open td/test-erasure-eio/0/block: (2) No such file or directory 2024-05-08T16:16:28.895 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:16:28.902+0000 7fb2288aa7c0 -1 bluestore(td/test-erasure-eio/0) _read_fsid unparsable uuid 2024-05-08T16:16:35.773 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/0/keyring 2024-05-08T16:16:35.774 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-05-08T16:16:35.774 INFO:tasks.workunit.client.0.smithi172.stdout:adding osd0 key to auth repository 2024-05-08T16:16:35.775 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-05-08T16:16:35.775 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-05-08T16:16:36.336 INFO:tasks.workunit.client.0.smithi172.stdout:start osd.0 2024-05-08T16:16:36.337 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-05-08T16:16:36.337 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=4457b0f1-028b-482b-8ee3-7ee05861e6cc --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81110/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-05-08T16:16:36.337 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-05-08T16:16:36.337 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-05-08T16:16:36.337 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-05-08T16:16:36.371 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:16:36.378+0000 7fa2929997c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:16:36.373 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:16:36.382+0000 7fa2929997c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:16:36.374 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:16:36.382+0000 7fa2929997c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:16:36.799 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-05-08T16:16:36.799 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-08T16:16:36.799 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-05-08T16:16:36.799 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-08T16:16:36.799 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-08T16:16:36.799 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:16:36.800 INFO:tasks.workunit.client.0.smithi172.stdout:0 2024-05-08T16:16:36.800 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-08T16:16:36.800 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:16:36.800 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:16:36.958 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:16:36.966+0000 7fa2929997c0 -1 Falling back to public interface 2024-05-08T16:16:37.217 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:16:38.219 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:16:38.219 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:16:38.219 INFO:tasks.workunit.client.0.smithi172.stdout:1 2024-05-08T16:16:38.219 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-08T16:16:38.220 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:16:38.220 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:16:38.622 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:16:39.623 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:16:39.623 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:16:39.624 INFO:tasks.workunit.client.0.smithi172.stdout:2 2024-05-08T16:16:39.624 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-08T16:16:39.624 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:16:39.624 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:16:39.732 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:16:39.738+0000 7fa2929997c0 -1 osd.0 0 log_to_monitors true 2024-05-08T16:16:40.057 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:16:41.058 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:16:41.059 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:16:41.059 INFO:tasks.workunit.client.0.smithi172.stdout:3 2024-05-08T16:16:41.059 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-05-08T16:16:41.059 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:16:41.060 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:16:41.578 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:16:42.580 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:16:42.580 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:16:42.581 INFO:tasks.workunit.client.0.smithi172.stdout:4 2024-05-08T16:16:42.581 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-05-08T16:16:42.581 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:16:42.581 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:16:43.034 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:16:44.035 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:16:44.035 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:16:44.036 INFO:tasks.workunit.client.0.smithi172.stdout:5 2024-05-08T16:16:44.037 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-05-08T16:16:44.037 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:16:44.037 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:16:44.570 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:16:45.572 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:16:45.573 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:16:45.573 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-05-08T16:16:45.573 INFO:tasks.workunit.client.0.smithi172.stdout:6 2024-05-08T16:16:45.574 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:16:45.574 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:16:46.022 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:16:47.023 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:16:47.024 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:16:47.024 INFO:tasks.workunit.client.0.smithi172.stdout:7 2024-05-08T16:16:47.025 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-05-08T16:16:47.025 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:16:47.025 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:16:47.607 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:16:48.608 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:16:48.608 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:16:48.609 INFO:tasks.workunit.client.0.smithi172.stdout:8 2024-05-08T16:16:48.609 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-05-08T16:16:48.609 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:16:48.609 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:16:49.066 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:16:50.069 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:16:50.069 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:16:50.069 INFO:tasks.workunit.client.0.smithi172.stdout:9 2024-05-08T16:16:50.070 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-05-08T16:16:50.070 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:16:50.070 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:16:50.629 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:16:51.630 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:16:51.630 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:16:51.630 INFO:tasks.workunit.client.0.smithi172.stdout:10 2024-05-08T16:16:51.631 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-05-08T16:16:51.631 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:16:51.631 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:16:52.182 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:16:53.184 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:16:53.184 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:16:53.184 INFO:tasks.workunit.client.0.smithi172.stdout:11 2024-05-08T16:16:53.185 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-05-08T16:16:53.185 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:16:53.185 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:16:54.415 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:16:55.416 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:16:55.416 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:16:55.417 INFO:tasks.workunit.client.0.smithi172.stdout:12 2024-05-08T16:16:55.417 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-05-08T16:16:55.417 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:16:55.417 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:16:56.337 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:16:57.339 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:16:57.339 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:16:57.339 INFO:tasks.workunit.client.0.smithi172.stdout:13 2024-05-08T16:16:57.340 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-05-08T16:16:57.340 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:16:57.340 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:16:58.174 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:16:59.175 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:16:59.175 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:16:59.175 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-05-08T16:16:59.175 INFO:tasks.workunit.client.0.smithi172.stdout:14 2024-05-08T16:16:59.176 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:16:59.176 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:17:00.183 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:17:01.184 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:17:01.184 INFO:tasks.workunit.client.0.smithi172.stdout:15 2024-05-08T16:17:01.185 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:17:01.185 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-05-08T16:17:01.185 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:17:01.185 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:17:02.241 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:17:03.242 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:17:03.243 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:17:03.243 INFO:tasks.workunit.client.0.smithi172.stdout:16 2024-05-08T16:17:03.243 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-05-08T16:17:03.243 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:17:03.243 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:17:04.146 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:17:05.148 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:17:05.148 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:17:05.148 INFO:tasks.workunit.client.0.smithi172.stdout:17 2024-05-08T16:17:05.149 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-05-08T16:17:05.149 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:17:05.149 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:17:05.653 INFO:tasks.workunit.client.0.smithi172.stdout:osd.0 up in weight 1 up_from 6 up_thru 6 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2705553667,v1:127.0.0.1:6803/2705553667] [v2:127.0.0.1:6804/2705553667,v1:127.0.0.1:6805/2705553667] exists,up a6dc0bdd-651f-4bcf-a3e2-3dcb3c3a9048 2024-05-08T16:17:05.657 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-08T16:17:05.657 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-08T16:17:05.657 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-08T16:17:05.657 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-05-08T16:17:05.657 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 1 2024-05-08T16:17:05.658 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-05-08T16:17:05.658 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-05-08T16:17:05.658 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-05-08T16:17:05.658 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-05-08T16:17:05.658 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/1 2024-05-08T16:17:05.658 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=4457b0f1-028b-482b-8ee3-7ee05861e6cc --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10' 2024-05-08T16:17:05.658 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-08T16:17:05.658 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-05-08T16:17:05.658 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-08T16:17:05.658 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/1' 2024-05-08T16:17:05.658 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/1/journal' 2024-05-08T16:17:05.659 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-05-08T16:17:05.659 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-05-08T16:17:05.659 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-05-08T16:17:05.659 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-05-08T16:17:05.659 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-08T16:17:05.659 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-08T16:17:05.659 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-08T16:17:05.659 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T16:17:05.659 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81110 2024-05-08T16:17:05.659 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.81110/$cluster-$name.asok' 2024-05-08T16:17:05.659 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.81110/$cluster-$name.asok' 2024-05-08T16:17:05.659 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-05-08T16:17:05.659 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-05-08T16:17:05.660 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-05-08T16:17:05.660 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-05-08T16:17:05.660 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-05-08T16:17:05.660 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-08T16:17:05.660 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-08T16:17:05.660 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-08T16:17:05.660 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-08T16:17:05.660 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-05-08T16:17:05.660 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-05-08T16:17:05.660 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/1 2024-05-08T16:17:05.661 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-05-08T16:17:05.661 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=f76c6d54-c852-4a9d-bfe7-0e539936499b 2024-05-08T16:17:05.661 INFO:tasks.workunit.client.0.smithi172.stdout:add osd1 f76c6d54-c852-4a9d-bfe7-0e539936499b 2024-05-08T16:17:05.663 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 f76c6d54-c852-4a9d-bfe7-0e539936499b' 2024-05-08T16:17:05.663 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-05-08T16:17:05.684 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCBpTtmadtWKRAATYePFdFavzk/niy1ZG+s3Q== 2024-05-08T16:17:05.685 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCBpTtmadtWKRAATYePFdFavzk/niy1ZG+s3Q=="}' 2024-05-08T16:17:05.685 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new f76c6d54-c852-4a9d-bfe7-0e539936499b -i td/test-erasure-eio/1/new.json 2024-05-08T16:17:06.155 INFO:tasks.workunit.client.0.smithi172.stdout:1 2024-05-08T16:17:06.171 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/1/new.json 2024-05-08T16:17:06.172 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=4457b0f1-028b-482b-8ee3-7ee05861e6cc --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81110/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCBpTtmadtWKRAATYePFdFavzk/niy1ZG+s3Q== --osd-uuid f76c6d54-c852-4a9d-bfe7-0e539936499b 2024-05-08T16:17:06.202 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:17:06.209+0000 7f92844c17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:17:06.250 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:17:06.257+0000 7f92844c17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:17:06.251 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:17:06.257+0000 7f92844c17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:17:06.252 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:17:06.257+0000 7f92844c17c0 -1 bluestore(td/test-erasure-eio/1/block) _read_bdev_label failed to open td/test-erasure-eio/1/block: (2) No such file or directory 2024-05-08T16:17:06.252 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:17:06.257+0000 7f92844c17c0 -1 bluestore(td/test-erasure-eio/1/block) _read_bdev_label failed to open td/test-erasure-eio/1/block: (2) No such file or directory 2024-05-08T16:17:06.252 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:17:06.257+0000 7f92844c17c0 -1 bluestore(td/test-erasure-eio/1/block) _read_bdev_label failed to open td/test-erasure-eio/1/block: (2) No such file or directory 2024-05-08T16:17:06.297 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:17:06.305+0000 7f92844c17c0 -1 bluestore(td/test-erasure-eio/1) _read_fsid unparsable uuid 2024-05-08T16:17:13.106 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/1/keyring 2024-05-08T16:17:13.107 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-05-08T16:17:13.108 INFO:tasks.workunit.client.0.smithi172.stdout:adding osd1 key to auth repository 2024-05-08T16:17:13.108 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-05-08T16:17:13.108 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-05-08T16:17:13.720 INFO:tasks.workunit.client.0.smithi172.stdout:start osd.1 2024-05-08T16:17:13.721 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-05-08T16:17:13.721 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=4457b0f1-028b-482b-8ee3-7ee05861e6cc --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81110/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-05-08T16:17:13.721 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-05-08T16:17:13.721 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-05-08T16:17:13.722 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-05-08T16:17:13.755 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:17:13.761+0000 7f6e971437c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:17:13.769 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:17:13.777+0000 7f6e971437c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:17:13.770 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:17:13.777+0000 7f6e971437c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:17:14.150 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-05-08T16:17:14.150 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-08T16:17:14.150 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-05-08T16:17:14.150 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-08T16:17:14.150 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-08T16:17:14.150 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:17:14.150 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-08T16:17:14.151 INFO:tasks.workunit.client.0.smithi172.stdout:0 2024-05-08T16:17:14.151 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:17:14.151 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T16:17:14.338 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:17:14.345+0000 7f6e971437c0 -1 Falling back to public interface 2024-05-08T16:17:14.561 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:17:15.563 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:17:15.563 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:17:15.563 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-08T16:17:15.563 INFO:tasks.workunit.client.0.smithi172.stdout:1 2024-05-08T16:17:15.564 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:17:15.564 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T16:17:15.990 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:17:16.935 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:17:16.941+0000 7f6e971437c0 -1 osd.1 0 log_to_monitors true 2024-05-08T16:17:16.991 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:17:16.991 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:17:16.991 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-08T16:17:16.992 INFO:tasks.workunit.client.0.smithi172.stdout:2 2024-05-08T16:17:16.992 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:17:16.992 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T16:17:17.694 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:17:18.695 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:17:18.695 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:17:18.696 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-05-08T16:17:18.696 INFO:tasks.workunit.client.0.smithi172.stdout:3 2024-05-08T16:17:18.696 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:17:18.696 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T16:17:19.126 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:17:20.128 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:17:20.128 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:17:20.128 INFO:tasks.workunit.client.0.smithi172.stdout:4 2024-05-08T16:17:20.129 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-05-08T16:17:20.129 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:17:20.129 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T16:17:20.584 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:17:21.586 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:17:21.586 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:17:21.586 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-05-08T16:17:21.586 INFO:tasks.workunit.client.0.smithi172.stdout:5 2024-05-08T16:17:21.587 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:17:21.587 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T16:17:22.114 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:17:23.116 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:17:23.116 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:17:23.116 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-05-08T16:17:23.116 INFO:tasks.workunit.client.0.smithi172.stdout:6 2024-05-08T16:17:23.116 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:17:23.117 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T16:17:23.543 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:17:24.545 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:17:24.545 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:17:24.545 INFO:tasks.workunit.client.0.smithi172.stdout:7 2024-05-08T16:17:24.546 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-05-08T16:17:24.546 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:17:24.546 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T16:17:25.042 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:17:26.044 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:17:26.044 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:17:26.044 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-05-08T16:17:26.044 INFO:tasks.workunit.client.0.smithi172.stdout:8 2024-05-08T16:17:26.044 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:17:26.044 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T16:17:26.592 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:17:27.594 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:17:27.594 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:17:27.594 INFO:tasks.workunit.client.0.smithi172.stdout:9 2024-05-08T16:17:27.595 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-05-08T16:17:27.595 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:17:27.595 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T16:17:28.287 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:17:29.289 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:17:29.289 INFO:tasks.workunit.client.0.smithi172.stdout:10 2024-05-08T16:17:29.290 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:17:29.290 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-05-08T16:17:29.290 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:17:29.290 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T16:17:29.940 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:17:30.942 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:17:30.942 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:17:30.943 INFO:tasks.workunit.client.0.smithi172.stdout:11 2024-05-08T16:17:30.943 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-05-08T16:17:30.943 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:17:30.944 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T16:17:31.440 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:17:32.441 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:17:32.441 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:17:32.442 INFO:tasks.workunit.client.0.smithi172.stdout:12 2024-05-08T16:17:32.442 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-05-08T16:17:32.442 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:17:32.442 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T16:17:33.205 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:17:34.207 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:17:34.207 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:17:34.208 INFO:tasks.workunit.client.0.smithi172.stdout:13 2024-05-08T16:17:34.208 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-05-08T16:17:34.208 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:17:34.208 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T16:17:35.496 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:17:36.497 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:17:36.498 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:17:36.498 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-05-08T16:17:36.498 INFO:tasks.workunit.client.0.smithi172.stdout:14 2024-05-08T16:17:36.498 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:17:36.498 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T16:17:37.307 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:17:38.308 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:17:38.309 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:17:38.309 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-05-08T16:17:38.309 INFO:tasks.workunit.client.0.smithi172.stdout:15 2024-05-08T16:17:38.309 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:17:38.309 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T16:17:39.382 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:17:40.385 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:17:40.385 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:17:40.385 INFO:tasks.workunit.client.0.smithi172.stdout:16 2024-05-08T16:17:40.385 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-05-08T16:17:40.386 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:17:40.386 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T16:17:41.502 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:17:42.504 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:17:42.504 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:17:42.504 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-05-08T16:17:42.505 INFO:tasks.workunit.client.0.smithi172.stdout:17 2024-05-08T16:17:42.505 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:17:42.505 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T16:17:42.972 INFO:tasks.workunit.client.0.smithi172.stdout:osd.1 up in weight 1 up_from 13 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/988232787,v1:127.0.0.1:6811/988232787] [v2:127.0.0.1:6812/988232787,v1:127.0.0.1:6813/988232787] exists,up f76c6d54-c852-4a9d-bfe7-0e539936499b 2024-05-08T16:17:42.975 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-08T16:17:42.975 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-08T16:17:42.975 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-08T16:17:42.975 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-05-08T16:17:42.975 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 2 2024-05-08T16:17:42.975 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-05-08T16:17:42.975 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-05-08T16:17:42.975 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-05-08T16:17:42.975 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-05-08T16:17:42.975 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/2 2024-05-08T16:17:42.976 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=4457b0f1-028b-482b-8ee3-7ee05861e6cc --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10' 2024-05-08T16:17:42.976 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-08T16:17:42.976 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-05-08T16:17:42.976 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-08T16:17:42.976 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/2' 2024-05-08T16:17:42.976 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/2/journal' 2024-05-08T16:17:42.976 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-05-08T16:17:42.976 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-05-08T16:17:42.976 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-05-08T16:17:42.976 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-05-08T16:17:42.976 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-08T16:17:42.976 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-08T16:17:42.976 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-08T16:17:42.976 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T16:17:42.977 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81110 2024-05-08T16:17:42.977 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.81110/$cluster-$name.asok' 2024-05-08T16:17:42.978 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.81110/$cluster-$name.asok' 2024-05-08T16:17:42.978 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-05-08T16:17:42.978 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-05-08T16:17:42.978 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-05-08T16:17:42.978 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-05-08T16:17:42.979 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-05-08T16:17:42.979 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-08T16:17:42.979 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-08T16:17:42.979 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-08T16:17:42.979 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-08T16:17:42.979 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-05-08T16:17:42.979 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-05-08T16:17:42.979 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/2 2024-05-08T16:17:42.979 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-05-08T16:17:42.980 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=91b6118c-0348-4a21-beef-0db38b0f7cce 2024-05-08T16:17:42.980 INFO:tasks.workunit.client.0.smithi172.stdout:add osd2 91b6118c-0348-4a21-beef-0db38b0f7cce 2024-05-08T16:17:42.980 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 91b6118c-0348-4a21-beef-0db38b0f7cce' 2024-05-08T16:17:42.981 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-05-08T16:17:43.000 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCnpTtmCtWVABAAPToBBp1f+wa8vJpIwz1Dbg== 2024-05-08T16:17:43.000 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCnpTtmCtWVABAAPToBBp1f+wa8vJpIwz1Dbg=="}' 2024-05-08T16:17:43.001 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 91b6118c-0348-4a21-beef-0db38b0f7cce -i td/test-erasure-eio/2/new.json 2024-05-08T16:17:43.813 INFO:tasks.workunit.client.0.smithi172.stdout:2 2024-05-08T16:17:43.831 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/2/new.json 2024-05-08T16:17:43.832 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=4457b0f1-028b-482b-8ee3-7ee05861e6cc --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81110/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCnpTtmCtWVABAAPToBBp1f+wa8vJpIwz1Dbg== --osd-uuid 91b6118c-0348-4a21-beef-0db38b0f7cce 2024-05-08T16:17:43.872 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:17:43.876+0000 7fb83e1257c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:17:43.874 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:17:43.880+0000 7fb83e1257c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:17:43.875 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:17:43.880+0000 7fb83e1257c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:17:43.876 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:17:43.880+0000 7fb83e1257c0 -1 bluestore(td/test-erasure-eio/2/block) _read_bdev_label failed to open td/test-erasure-eio/2/block: (2) No such file or directory 2024-05-08T16:17:43.876 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:17:43.880+0000 7fb83e1257c0 -1 bluestore(td/test-erasure-eio/2/block) _read_bdev_label failed to open td/test-erasure-eio/2/block: (2) No such file or directory 2024-05-08T16:17:43.876 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:17:43.880+0000 7fb83e1257c0 -1 bluestore(td/test-erasure-eio/2/block) _read_bdev_label failed to open td/test-erasure-eio/2/block: (2) No such file or directory 2024-05-08T16:17:43.960 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:17:43.964+0000 7fb83e1257c0 -1 bluestore(td/test-erasure-eio/2) _read_fsid unparsable uuid 2024-05-08T16:17:50.818 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/2/keyring 2024-05-08T16:17:50.818 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-05-08T16:17:50.819 INFO:tasks.workunit.client.0.smithi172.stdout:adding osd2 key to auth repository 2024-05-08T16:17:50.819 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-05-08T16:17:50.819 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-05-08T16:17:51.385 INFO:tasks.workunit.client.0.smithi172.stdout:start osd.2 2024-05-08T16:17:51.386 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-05-08T16:17:51.386 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=4457b0f1-028b-482b-8ee3-7ee05861e6cc --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81110/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-05-08T16:17:51.386 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-05-08T16:17:51.386 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-05-08T16:17:51.386 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-05-08T16:17:51.415 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:17:51.420+0000 7fe4948517c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:17:51.446 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:17:51.452+0000 7fe4948517c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:17:51.448 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:17:51.452+0000 7fe4948517c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:17:51.808 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-05-08T16:17:51.808 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-08T16:17:51.808 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-05-08T16:17:51.809 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-08T16:17:51.809 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-08T16:17:51.809 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:17:51.809 INFO:tasks.workunit.client.0.smithi172.stdout:0 2024-05-08T16:17:51.809 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-08T16:17:51.810 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:17:51.810 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T16:17:52.015 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:17:52.020+0000 7fe4948517c0 -1 Falling back to public interface 2024-05-08T16:17:52.216 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:17:53.218 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:17:53.218 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:17:53.218 INFO:tasks.workunit.client.0.smithi172.stdout:1 2024-05-08T16:17:53.219 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-08T16:17:53.219 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:17:53.219 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T16:17:53.625 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:17:54.627 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:17:54.627 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:17:54.628 INFO:tasks.workunit.client.0.smithi172.stdout:2 2024-05-08T16:17:54.628 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-08T16:17:54.628 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:17:54.628 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T16:17:54.697 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:17:54.700+0000 7fe4948517c0 -1 osd.2 0 log_to_monitors true 2024-05-08T16:17:55.072 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:17:56.074 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:17:56.074 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:17:56.074 INFO:tasks.workunit.client.0.smithi172.stdout:3 2024-05-08T16:17:56.074 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-05-08T16:17:56.075 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:17:56.075 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T16:17:56.676 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:17:57.678 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:17:57.678 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:17:57.679 INFO:tasks.workunit.client.0.smithi172.stdout:4 2024-05-08T16:17:57.679 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-05-08T16:17:57.679 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:17:57.679 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T16:17:58.177 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:17:59.179 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:17:59.180 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:17:59.180 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-05-08T16:17:59.180 INFO:tasks.workunit.client.0.smithi172.stdout:5 2024-05-08T16:17:59.180 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:17:59.180 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T16:17:59.727 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:18:00.729 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:18:00.730 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:18:00.730 INFO:tasks.workunit.client.0.smithi172.stdout:6 2024-05-08T16:18:00.730 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-05-08T16:18:00.730 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:18:00.730 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T16:18:01.185 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:18:02.186 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:18:02.187 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:18:02.187 INFO:tasks.workunit.client.0.smithi172.stdout:7 2024-05-08T16:18:02.187 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-05-08T16:18:02.187 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:18:02.187 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T16:18:02.660 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:18:03.662 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:18:03.662 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:18:03.662 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-05-08T16:18:03.663 INFO:tasks.workunit.client.0.smithi172.stdout:8 2024-05-08T16:18:03.663 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:18:03.663 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T16:18:04.115 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:18:05.117 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:18:05.117 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:18:05.117 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-05-08T16:18:05.117 INFO:tasks.workunit.client.0.smithi172.stdout:9 2024-05-08T16:18:05.118 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:18:05.118 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T16:18:05.663 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:18:06.664 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:18:06.664 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:18:06.665 INFO:tasks.workunit.client.0.smithi172.stdout:10 2024-05-08T16:18:06.665 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-05-08T16:18:06.665 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:18:06.665 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T16:18:07.119 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:18:08.121 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:18:08.122 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:18:08.122 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-05-08T16:18:08.122 INFO:tasks.workunit.client.0.smithi172.stdout:11 2024-05-08T16:18:08.123 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:18:08.123 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T16:18:09.368 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:18:10.370 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:18:10.370 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:18:10.370 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-05-08T16:18:10.371 INFO:tasks.workunit.client.0.smithi172.stdout:12 2024-05-08T16:18:10.371 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:18:10.371 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T16:18:11.208 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:18:12.210 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:18:12.210 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:18:12.210 INFO:tasks.workunit.client.0.smithi172.stdout:13 2024-05-08T16:18:12.211 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-05-08T16:18:12.211 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:18:12.211 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T16:18:12.935 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:18:13.937 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:18:13.937 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:18:13.938 INFO:tasks.workunit.client.0.smithi172.stdout:14 2024-05-08T16:18:13.938 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-05-08T16:18:13.938 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:18:13.938 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T16:18:15.196 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:18:16.198 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:18:16.198 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:18:16.198 INFO:tasks.workunit.client.0.smithi172.stdout:15 2024-05-08T16:18:16.198 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-05-08T16:18:16.199 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:18:16.199 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T16:18:17.016 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:18:18.018 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:18:18.018 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:18:18.019 INFO:tasks.workunit.client.0.smithi172.stdout:16 2024-05-08T16:18:18.019 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-05-08T16:18:18.019 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:18:18.019 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T16:18:18.715 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:18:19.717 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:18:19.717 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:18:19.718 INFO:tasks.workunit.client.0.smithi172.stdout:17 2024-05-08T16:18:19.718 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-05-08T16:18:19.718 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:18:19.718 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T16:18:20.256 INFO:tasks.workunit.client.0.smithi172.stdout:osd.2 up in weight 1 up_from 18 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/1405192275,v1:127.0.0.1:6819/1405192275] [v2:127.0.0.1:6820/1405192275,v1:127.0.0.1:6821/1405192275] exists,up 91b6118c-0348-4a21-beef-0db38b0f7cce 2024-05-08T16:18:20.261 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-08T16:18:20.261 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-08T16:18:20.261 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-08T16:18:20.261 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-05-08T16:18:20.261 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 3 2024-05-08T16:18:20.261 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-05-08T16:18:20.261 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-05-08T16:18:20.261 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2024-05-08T16:18:20.261 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-05-08T16:18:20.262 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/3 2024-05-08T16:18:20.262 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=4457b0f1-028b-482b-8ee3-7ee05861e6cc --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10' 2024-05-08T16:18:20.262 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-08T16:18:20.262 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-05-08T16:18:20.262 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-08T16:18:20.262 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/3' 2024-05-08T16:18:20.262 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/3/journal' 2024-05-08T16:18:20.262 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-05-08T16:18:20.262 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-05-08T16:18:20.262 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-05-08T16:18:20.262 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-05-08T16:18:20.262 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-08T16:18:20.262 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-08T16:18:20.263 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-08T16:18:20.263 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T16:18:20.263 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81110 2024-05-08T16:18:20.263 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.81110/$cluster-$name.asok' 2024-05-08T16:18:20.263 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.81110/$cluster-$name.asok' 2024-05-08T16:18:20.263 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-05-08T16:18:20.263 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-05-08T16:18:20.263 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-05-08T16:18:20.263 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-05-08T16:18:20.263 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-05-08T16:18:20.263 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-08T16:18:20.263 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-08T16:18:20.264 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-08T16:18:20.264 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-08T16:18:20.264 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-05-08T16:18:20.264 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-05-08T16:18:20.264 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/3 2024-05-08T16:18:20.264 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-05-08T16:18:20.264 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=89162b94-da89-468a-9089-65378adb80a3 2024-05-08T16:18:20.265 INFO:tasks.workunit.client.0.smithi172.stdout:add osd3 89162b94-da89-468a-9089-65378adb80a3 2024-05-08T16:18:20.265 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 89162b94-da89-468a-9089-65378adb80a3' 2024-05-08T16:18:20.265 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-05-08T16:18:20.288 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDMpTtmzBezERAArpjEu5VnvJiMJ+hxEtQfeQ== 2024-05-08T16:18:20.288 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDMpTtmzBezERAArpjEu5VnvJiMJ+hxEtQfeQ=="}' 2024-05-08T16:18:20.288 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 89162b94-da89-468a-9089-65378adb80a3 -i td/test-erasure-eio/3/new.json 2024-05-08T16:18:20.771 INFO:tasks.workunit.client.0.smithi172.stdout:3 2024-05-08T16:18:20.787 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/3/new.json 2024-05-08T16:18:20.788 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=4457b0f1-028b-482b-8ee3-7ee05861e6cc --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81110/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDMpTtmzBezERAArpjEu5VnvJiMJ+hxEtQfeQ== --osd-uuid 89162b94-da89-468a-9089-65378adb80a3 2024-05-08T16:18:20.827 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:18:20.835+0000 7f1050d7e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:18:21.016 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:18:21.023+0000 7f1050d7e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:18:21.018 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:18:21.027+0000 7f1050d7e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:18:21.018 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:18:21.027+0000 7f1050d7e7c0 -1 bluestore(td/test-erasure-eio/3/block) _read_bdev_label failed to open td/test-erasure-eio/3/block: (2) No such file or directory 2024-05-08T16:18:21.018 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:18:21.027+0000 7f1050d7e7c0 -1 bluestore(td/test-erasure-eio/3/block) _read_bdev_label failed to open td/test-erasure-eio/3/block: (2) No such file or directory 2024-05-08T16:18:21.018 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:18:21.027+0000 7f1050d7e7c0 -1 bluestore(td/test-erasure-eio/3/block) _read_bdev_label failed to open td/test-erasure-eio/3/block: (2) No such file or directory 2024-05-08T16:18:21.167 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:18:21.175+0000 7f1050d7e7c0 -1 bluestore(td/test-erasure-eio/3) _read_fsid unparsable uuid 2024-05-08T16:18:28.335 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/3/keyring 2024-05-08T16:18:28.336 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-05-08T16:18:28.336 INFO:tasks.workunit.client.0.smithi172.stdout:adding osd3 key to auth repository 2024-05-08T16:18:28.336 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2024-05-08T16:18:28.337 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-05-08T16:18:28.895 INFO:tasks.workunit.client.0.smithi172.stdout:start osd.3 2024-05-08T16:18:28.897 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2024-05-08T16:18:28.897 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-05-08T16:18:28.897 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-05-08T16:18:28.897 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=4457b0f1-028b-482b-8ee3-7ee05861e6cc --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81110/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-05-08T16:18:28.898 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-05-08T16:18:28.933 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:18:28.939+0000 7f69a8f887c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:18:28.955 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:18:28.963+0000 7f69a8f887c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:18:28.957 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:18:28.963+0000 7f69a8f887c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:18:29.324 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2024-05-08T16:18:29.324 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-08T16:18:29.324 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-05-08T16:18:29.324 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-08T16:18:29.324 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-08T16:18:29.324 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:18:29.324 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-08T16:18:29.324 INFO:tasks.workunit.client.0.smithi172.stdout:0 2024-05-08T16:18:29.325 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:18:29.325 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T16:18:29.515 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:18:29.523+0000 7f69a8f887c0 -1 Falling back to public interface 2024-05-08T16:18:29.936 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:18:30.938 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:18:30.938 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:18:30.938 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-08T16:18:30.938 INFO:tasks.workunit.client.0.smithi172.stdout:1 2024-05-08T16:18:30.939 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:18:30.939 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T16:18:31.356 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:18:32.352 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:18:32.359+0000 7f69a8f887c0 -1 osd.3 0 log_to_monitors true 2024-05-08T16:18:32.357 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:18:32.357 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:18:32.357 INFO:tasks.workunit.client.0.smithi172.stdout:2 2024-05-08T16:18:32.358 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-08T16:18:32.358 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:18:32.358 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T16:18:32.901 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:18:33.902 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:18:33.903 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:18:33.903 INFO:tasks.workunit.client.0.smithi172.stdout:3 2024-05-08T16:18:33.903 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-05-08T16:18:33.904 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:18:33.904 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T16:18:34.575 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:18:35.576 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:18:35.577 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:18:35.577 INFO:tasks.workunit.client.0.smithi172.stdout:4 2024-05-08T16:18:35.577 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-05-08T16:18:35.577 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:18:35.578 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T16:18:36.118 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:18:37.121 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:18:37.121 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:18:37.121 INFO:tasks.workunit.client.0.smithi172.stdout:5 2024-05-08T16:18:37.122 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-05-08T16:18:37.122 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:18:37.122 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T16:18:37.655 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:18:38.657 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:18:38.657 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:18:38.658 INFO:tasks.workunit.client.0.smithi172.stdout:6 2024-05-08T16:18:38.658 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-05-08T16:18:38.658 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:18:38.658 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T16:18:39.232 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:18:40.233 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:18:40.234 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:18:40.234 INFO:tasks.workunit.client.0.smithi172.stdout:7 2024-05-08T16:18:40.234 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-05-08T16:18:40.235 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:18:40.235 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T16:18:40.700 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:18:41.702 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:18:41.702 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:18:41.702 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-05-08T16:18:41.702 INFO:tasks.workunit.client.0.smithi172.stdout:8 2024-05-08T16:18:41.703 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:18:41.703 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T16:18:42.228 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:18:43.229 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:18:43.229 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:18:43.230 INFO:tasks.workunit.client.0.smithi172.stdout:9 2024-05-08T16:18:43.230 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-05-08T16:18:43.230 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:18:43.230 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T16:18:43.662 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:18:44.664 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:18:44.664 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:18:44.664 INFO:tasks.workunit.client.0.smithi172.stdout:10 2024-05-08T16:18:44.665 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-05-08T16:18:44.665 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:18:44.665 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T16:18:45.244 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:18:46.245 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:18:46.245 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:18:46.246 INFO:tasks.workunit.client.0.smithi172.stdout:11 2024-05-08T16:18:46.246 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-05-08T16:18:46.246 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:18:46.246 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T16:18:47.073 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:18:48.074 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:18:48.075 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:18:48.075 INFO:tasks.workunit.client.0.smithi172.stdout:12 2024-05-08T16:18:48.075 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-05-08T16:18:48.075 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:18:48.076 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T16:18:48.957 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:18:49.958 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:18:49.959 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:18:49.959 INFO:tasks.workunit.client.0.smithi172.stdout:13 2024-05-08T16:18:49.959 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-05-08T16:18:49.959 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:18:49.960 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T16:18:50.780 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:18:51.782 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:18:51.782 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:18:51.782 INFO:tasks.workunit.client.0.smithi172.stdout:14 2024-05-08T16:18:51.783 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-05-08T16:18:51.783 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:18:51.783 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T16:18:52.734 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:18:53.736 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:18:53.736 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:18:53.736 INFO:tasks.workunit.client.0.smithi172.stdout:15 2024-05-08T16:18:53.737 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-05-08T16:18:53.737 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:18:53.737 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T16:18:54.614 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:18:55.616 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:18:55.616 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:18:55.617 INFO:tasks.workunit.client.0.smithi172.stdout:16 2024-05-08T16:18:55.617 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-05-08T16:18:55.617 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:18:55.617 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T16:18:57.093 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:18:58.095 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:18:58.095 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:18:58.095 INFO:tasks.workunit.client.0.smithi172.stdout:17 2024-05-08T16:18:58.096 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-05-08T16:18:58.096 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:18:58.096 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T16:18:58.767 INFO:tasks.workunit.client.0.smithi172.stdout:osd.3 up in weight 1 up_from 27 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/2429615901,v1:127.0.0.1:6827/2429615901] [v2:127.0.0.1:6828/2429615901,v1:127.0.0.1:6829/2429615901] exists,up 89162b94-da89-468a-9089-65378adb80a3 2024-05-08T16:18:58.771 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-08T16:18:58.771 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-08T16:18:58.771 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-08T16:18:58.771 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-05-08T16:18:58.771 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 4 2024-05-08T16:18:58.771 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-05-08T16:18:58.771 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-05-08T16:18:58.771 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=4 2024-05-08T16:18:58.771 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-05-08T16:18:58.771 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/4 2024-05-08T16:18:58.771 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=4457b0f1-028b-482b-8ee3-7ee05861e6cc --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10' 2024-05-08T16:18:58.771 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-08T16:18:58.772 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-05-08T16:18:58.772 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-08T16:18:58.772 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/4' 2024-05-08T16:18:58.772 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/4/journal' 2024-05-08T16:18:58.772 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-05-08T16:18:58.772 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-05-08T16:18:58.772 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-05-08T16:18:58.772 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-05-08T16:18:58.772 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-08T16:18:58.773 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-08T16:18:58.773 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-08T16:18:58.773 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T16:18:58.773 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81110 2024-05-08T16:18:58.773 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.81110/$cluster-$name.asok' 2024-05-08T16:18:58.773 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.81110/$cluster-$name.asok' 2024-05-08T16:18:58.773 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-05-08T16:18:58.773 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-05-08T16:18:58.773 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-05-08T16:18:58.773 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-05-08T16:18:58.773 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-05-08T16:18:58.773 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-08T16:18:58.773 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-08T16:18:58.773 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-08T16:18:58.773 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-08T16:18:58.774 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-05-08T16:18:58.774 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-05-08T16:18:58.774 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/4 2024-05-08T16:18:58.774 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-05-08T16:18:58.774 INFO:tasks.workunit.client.0.smithi172.stdout:add osd4 07d1f8d3-bf08-4d9b-a07b-3d007a15bc57 2024-05-08T16:18:58.775 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=07d1f8d3-bf08-4d9b-a07b-3d007a15bc57 2024-05-08T16:18:58.775 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd4 07d1f8d3-bf08-4d9b-a07b-3d007a15bc57' 2024-05-08T16:18:58.775 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-05-08T16:18:58.792 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDypTtm6QLGLxAAWG90gVoDQrCgiDzpZ+mjeA== 2024-05-08T16:18:58.792 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDypTtm6QLGLxAAWG90gVoDQrCgiDzpZ+mjeA=="}' 2024-05-08T16:18:58.792 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 07d1f8d3-bf08-4d9b-a07b-3d007a15bc57 -i td/test-erasure-eio/4/new.json 2024-05-08T16:18:59.479 INFO:tasks.workunit.client.0.smithi172.stdout:4 2024-05-08T16:18:59.496 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/4/new.json 2024-05-08T16:18:59.497 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 4 --fsid=4457b0f1-028b-482b-8ee3-7ee05861e6cc --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/4 --osd-journal=td/test-erasure-eio/4/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81110/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDypTtm6QLGLxAAWG90gVoDQrCgiDzpZ+mjeA== --osd-uuid 07d1f8d3-bf08-4d9b-a07b-3d007a15bc57 2024-05-08T16:18:59.530 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:18:59.538+0000 7fb6c22057c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:18:59.532 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:18:59.538+0000 7fb6c22057c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:18:59.533 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:18:59.542+0000 7fb6c22057c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:18:59.533 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:18:59.542+0000 7fb6c22057c0 -1 bluestore(td/test-erasure-eio/4/block) _read_bdev_label failed to open td/test-erasure-eio/4/block: (2) No such file or directory 2024-05-08T16:18:59.534 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:18:59.542+0000 7fb6c22057c0 -1 bluestore(td/test-erasure-eio/4/block) _read_bdev_label failed to open td/test-erasure-eio/4/block: (2) No such file or directory 2024-05-08T16:18:59.534 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:18:59.542+0000 7fb6c22057c0 -1 bluestore(td/test-erasure-eio/4/block) _read_bdev_label failed to open td/test-erasure-eio/4/block: (2) No such file or directory 2024-05-08T16:18:59.666 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:18:59.670+0000 7fb6c22057c0 -1 bluestore(td/test-erasure-eio/4) _read_fsid unparsable uuid 2024-05-08T16:19:06.942 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/4/keyring 2024-05-08T16:19:06.942 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-05-08T16:19:06.943 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd4 key to auth repository 2024-05-08T16:19:06.943 INFO:tasks.workunit.client.0.smithi172.stdout:adding osd4 key to auth repository 2024-05-08T16:19:06.944 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-05-08T16:19:07.504 INFO:tasks.workunit.client.0.smithi172.stdout:start osd.4 2024-05-08T16:19:07.505 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.4 2024-05-08T16:19:07.505 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 4 --fsid=4457b0f1-028b-482b-8ee3-7ee05861e6cc --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/4 --osd-journal=td/test-erasure-eio/4/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81110/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-05-08T16:19:07.505 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-05-08T16:19:07.505 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-05-08T16:19:07.506 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-05-08T16:19:07.541 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:19:07.550+0000 7f97da87b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:19:07.583 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:19:07.590+0000 7f97da87b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:19:07.585 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:19:07.590+0000 7f97da87b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:19:07.981 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 4 2024-05-08T16:19:07.981 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-08T16:19:07.981 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=4 2024-05-08T16:19:07.981 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-08T16:19:07.981 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-08T16:19:07.981 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:19:07.981 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-08T16:19:07.981 INFO:tasks.workunit.client.0.smithi172.stdout:0 2024-05-08T16:19:07.982 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:19:07.982 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-08T16:19:08.148 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:19:08.154+0000 7f97da87b7c0 -1 Falling back to public interface 2024-05-08T16:19:08.397 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:19:09.398 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:19:09.398 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:19:09.399 INFO:tasks.workunit.client.0.smithi172.stdout:1 2024-05-08T16:19:09.399 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-08T16:19:09.399 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:19:09.399 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-08T16:19:09.835 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:19:10.837 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:19:10.837 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:19:10.837 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-08T16:19:10.837 INFO:tasks.workunit.client.0.smithi172.stdout:2 2024-05-08T16:19:10.838 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:19:10.838 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-08T16:19:10.839 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:19:10.846+0000 7f97da87b7c0 -1 osd.4 0 log_to_monitors true 2024-05-08T16:19:11.398 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:19:12.400 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:19:12.400 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:19:12.400 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-05-08T16:19:12.400 INFO:tasks.workunit.client.0.smithi172.stdout:3 2024-05-08T16:19:12.401 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:19:12.401 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-08T16:19:12.993 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:19:13.995 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:19:13.995 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:19:13.995 INFO:tasks.workunit.client.0.smithi172.stdout:4 2024-05-08T16:19:13.996 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-05-08T16:19:13.996 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:19:13.996 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-08T16:19:14.652 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:19:15.654 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:19:15.654 INFO:tasks.workunit.client.0.smithi172.stdout:5 2024-05-08T16:19:15.655 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:19:15.655 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-05-08T16:19:15.655 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:19:15.655 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-08T16:19:16.372 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:19:17.373 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:19:17.374 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:19:17.374 INFO:tasks.workunit.client.0.smithi172.stdout:6 2024-05-08T16:19:17.375 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-05-08T16:19:17.375 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:19:17.375 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-08T16:19:18.010 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:19:19.012 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:19:19.012 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:19:19.012 INFO:tasks.workunit.client.0.smithi172.stdout:7 2024-05-08T16:19:19.013 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-05-08T16:19:19.013 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:19:19.013 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-08T16:19:19.433 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:19:20.435 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:19:20.436 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:19:20.436 INFO:tasks.workunit.client.0.smithi172.stdout:8 2024-05-08T16:19:20.437 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-05-08T16:19:20.437 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:19:20.438 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-08T16:19:20.878 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:19:21.879 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:19:21.880 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:19:21.880 INFO:tasks.workunit.client.0.smithi172.stdout:9 2024-05-08T16:19:21.880 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-05-08T16:19:21.880 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:19:21.881 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-08T16:19:22.353 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:19:23.354 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:19:23.354 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:19:23.355 INFO:tasks.workunit.client.0.smithi172.stdout:10 2024-05-08T16:19:23.356 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-05-08T16:19:23.356 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:19:23.356 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-08T16:19:23.858 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:19:24.860 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:19:24.860 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:19:24.860 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-05-08T16:19:24.860 INFO:tasks.workunit.client.0.smithi172.stdout:11 2024-05-08T16:19:24.860 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:19:24.861 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-08T16:19:25.279 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:19:26.281 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:19:26.281 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:19:26.281 INFO:tasks.workunit.client.0.smithi172.stdout:12 2024-05-08T16:19:26.282 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-05-08T16:19:26.282 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:19:26.282 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-08T16:19:26.908 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:19:27.910 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:19:27.910 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:19:27.910 INFO:tasks.workunit.client.0.smithi172.stdout:13 2024-05-08T16:19:27.911 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-05-08T16:19:27.911 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:19:27.911 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-08T16:19:28.770 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:19:29.772 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:19:29.772 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:19:29.772 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-05-08T16:19:29.772 INFO:tasks.workunit.client.0.smithi172.stdout:14 2024-05-08T16:19:29.773 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:19:29.773 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-08T16:19:30.271 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:19:31.273 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:19:31.273 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:19:31.273 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-05-08T16:19:31.273 INFO:tasks.workunit.client.0.smithi172.stdout:15 2024-05-08T16:19:31.274 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:19:31.274 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-08T16:19:32.093 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:19:33.094 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:19:33.094 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:19:33.095 INFO:tasks.workunit.client.0.smithi172.stdout:16 2024-05-08T16:19:33.095 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-05-08T16:19:33.095 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:19:33.095 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-08T16:19:33.957 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:19:34.959 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:19:34.959 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:19:34.959 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-05-08T16:19:34.959 INFO:tasks.workunit.client.0.smithi172.stdout:17 2024-05-08T16:19:34.960 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:19:34.960 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-08T16:19:35.687 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:19:36.689 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:19:36.689 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:19:36.690 INFO:tasks.workunit.client.0.smithi172.stdout:18 2024-05-08T16:19:36.690 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-05-08T16:19:36.691 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:19:36.691 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-08T16:19:37.149 INFO:tasks.workunit.client.0.smithi172.stdout:osd.4 up in weight 1 up_from 36 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/1210356610,v1:127.0.0.1:6835/1210356610] [v2:127.0.0.1:6836/1210356610,v1:127.0.0.1:6837/1210356610] exists,up 07d1f8d3-bf08-4d9b-a07b-3d007a15bc57 2024-05-08T16:19:37.149 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-08T16:19:37.149 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-08T16:19:37.150 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-08T16:19:37.151 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: get_asok_path osd.0 2024-05-08T16:19:37.151 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-05-08T16:19:37.151 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-05-08T16:19:37.151 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-08T16:19:37.151 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T16:19:37.151 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81110 2024-05-08T16:19:37.152 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.81110/ceph-osd.0.asok 2024-05-08T16:19:37.152 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: CEPH_ARGS= 2024-05-08T16:19:37.152 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: ceph --admin-daemon /tmp/ceph-asok.81110/ceph-osd.0.asok log flush 2024-05-08T16:19:37.260 INFO:tasks.workunit.client.0.smithi172.stdout:{} 2024-05-08T16:19:37.273 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:56: setup_osds: grep 'load: jerasure.*lrc' td/test-erasure-eio/osd.0.log 2024-05-08T16:19:37.278 INFO:tasks.workunit.client.0.smithi172.stdout:2024-05-08T16:16:36.970+0000 7fa2929997c0 0 load: jerasure load: lrc 2024-05-08T16:19:37.279 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:520: TEST_ec_backfill_unfound: CEPH_ARGS='--fsid=4457b0f1-028b-482b-8ee3-7ee05861e6cc --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-05-08T16:19:37.279 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:522: TEST_ec_backfill_unfound: local poolname=pool-jerasure 2024-05-08T16:19:37.279 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:523: TEST_ec_backfill_unfound: create_erasure_coded_pool pool-jerasure 3 2 2024-05-08T16:19:37.279 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:67: create_erasure_coded_pool: local poolname=pool-jerasure 2024-05-08T16:19:37.279 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:68: create_erasure_coded_pool: shift 2024-05-08T16:19:37.279 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:69: create_erasure_coded_pool: local k=3 2024-05-08T16:19:37.279 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:70: create_erasure_coded_pool: shift 2024-05-08T16:19:37.279 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:71: create_erasure_coded_pool: local m=2 2024-05-08T16:19:37.279 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:72: create_erasure_coded_pool: shift 2024-05-08T16:19:37.279 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:74: create_erasure_coded_pool: ceph osd erasure-code-profile set myprofile plugin=jerasure k=3 m=2 crush-failure-domain=osd 2024-05-08T16:19:37.868 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:78: create_erasure_coded_pool: create_pool pool-jerasure 1 1 erasure myprofile 2024-05-08T16:19:37.868 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create pool-jerasure 1 1 erasure myprofile 2024-05-08T16:19:39.472 INFO:tasks.workunit.client.0.smithi172.stderr:pool 'pool-jerasure' already exists 2024-05-08T16:19:39.495 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-05-08T16:19:40.497 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:80: create_erasure_coded_pool: wait_for_clean 2024-05-08T16:19:40.497 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-05-08T16:19:40.497 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-05-08T16:19:40.497 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-05-08T16:19:40.498 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-05-08T16:19:40.498 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-05-08T16:19:40.498 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-05-08T16:19:40.499 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-05-08T16:19:40.499 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-05-08T16:19:40.499 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-05-08T16:19:40.622 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-05-08T16:19:40.623 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-05-08T16:19:40.623 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-05-08T16:19:40.623 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-05-08T16:19:40.623 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-05-08T16:19:40.623 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-05-08T16:19:41.053 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-05-08T16:19:41.054 INFO:tasks.workunit.client.0.smithi172.stderr:1 2024-05-08T16:19:41.054 INFO:tasks.workunit.client.0.smithi172.stderr:2 2024-05-08T16:19:41.054 INFO:tasks.workunit.client.0.smithi172.stderr:3 2024-05-08T16:19:41.054 INFO:tasks.workunit.client.0.smithi172.stderr:4' 2024-05-08T16:19:41.054 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-05-08T16:19:41.054 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T16:19:41.054 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-05-08T16:19:41.351 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=25769803817 2024-05-08T16:19:41.352 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 25769803817 2024-05-08T16:19:41.352 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803817' 2024-05-08T16:19:41.352 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T16:19:41.352 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-05-08T16:19:41.509 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=55834574875 2024-05-08T16:19:41.509 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 55834574875 2024-05-08T16:19:41.509 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803817 1-55834574875' 2024-05-08T16:19:41.509 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T16:19:41.510 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-05-08T16:19:41.696 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=77309411353 2024-05-08T16:19:41.696 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 77309411353 2024-05-08T16:19:41.696 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803817 1-55834574875 2-77309411353' 2024-05-08T16:19:41.696 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T16:19:41.696 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-05-08T16:19:41.849 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=115964117007 2024-05-08T16:19:41.850 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 115964117007 2024-05-08T16:19:41.850 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803817 1-55834574875 2-77309411353 3-115964117007' 2024-05-08T16:19:41.850 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T16:19:41.850 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-05-08T16:19:42.001 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=154618822665 2024-05-08T16:19:42.001 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 154618822665 2024-05-08T16:19:42.001 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803817 1-55834574875 2-77309411353 3-115964117007 4-154618822665' 2024-05-08T16:19:42.001 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-08T16:19:42.002 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-25769803817 2024-05-08T16:19:42.002 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-08T16:19:42.002 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-05-08T16:19:42.004 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-25769803817 2024-05-08T16:19:42.004 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-08T16:19:42.005 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=25769803817 2024-05-08T16:19:42.005 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 25769803817' 2024-05-08T16:19:42.005 INFO:tasks.workunit.client.0.smithi172.stdout:waiting osd.0 seq 25769803817 2024-05-08T16:19:42.006 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-05-08T16:19:42.424 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 25769803817 -lt 25769803817 2024-05-08T16:19:42.424 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-08T16:19:42.425 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-55834574875 2024-05-08T16:19:42.425 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-08T16:19:42.426 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-05-08T16:19:42.427 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-55834574875 2024-05-08T16:19:42.427 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-08T16:19:42.428 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=55834574875 2024-05-08T16:19:42.428 INFO:tasks.workunit.client.0.smithi172.stdout:waiting osd.1 seq 55834574875 2024-05-08T16:19:42.428 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 55834574875' 2024-05-08T16:19:42.429 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-05-08T16:19:42.893 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 55834574875 -lt 55834574875 2024-05-08T16:19:42.893 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-08T16:19:42.894 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-77309411353 2024-05-08T16:19:42.894 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-08T16:19:42.896 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-05-08T16:19:42.897 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-77309411353 2024-05-08T16:19:42.897 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-08T16:19:42.898 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=77309411353 2024-05-08T16:19:42.898 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 77309411353' 2024-05-08T16:19:42.899 INFO:tasks.workunit.client.0.smithi172.stdout:waiting osd.2 seq 77309411353 2024-05-08T16:19:42.899 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-05-08T16:19:43.299 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 77309411351 -lt 77309411353 2024-05-08T16:19:43.299 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-05-08T16:19:44.300 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-05-08T16:19:44.301 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-05-08T16:19:44.721 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 77309411353 -lt 77309411353 2024-05-08T16:19:44.721 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-08T16:19:44.722 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-08T16:19:44.722 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-115964117007 2024-05-08T16:19:44.723 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-05-08T16:19:44.724 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-115964117007 2024-05-08T16:19:44.724 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-08T16:19:44.725 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=115964117007 2024-05-08T16:19:44.725 INFO:tasks.workunit.client.0.smithi172.stdout:waiting osd.3 seq 115964117007 2024-05-08T16:19:44.726 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 115964117007' 2024-05-08T16:19:44.726 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-05-08T16:19:45.140 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 115964117007 -lt 115964117007 2024-05-08T16:19:45.141 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-08T16:19:45.142 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-154618822665 2024-05-08T16:19:45.142 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-08T16:19:45.143 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-05-08T16:19:45.144 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-154618822665 2024-05-08T16:19:45.144 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-08T16:19:45.145 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=154618822665 2024-05-08T16:19:45.145 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 154618822665' 2024-05-08T16:19:45.145 INFO:tasks.workunit.client.0.smithi172.stdout:waiting osd.4 seq 154618822665 2024-05-08T16:19:45.145 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-05-08T16:19:45.568 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 154618822666 -lt 154618822665 2024-05-08T16:19:45.569 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-05-08T16:19:45.569 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-05-08T16:19:45.570 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-05-08T16:19:46.082 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-05-08T16:19:46.082 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-05-08T16:19:46.083 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-05-08T16:19:46.083 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-05-08T16:19:46.083 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-05-08T16:19:46.083 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-05-08T16:19:46.083 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-05-08T16:19:46.084 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-05-08T16:19:46.513 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-05-08T16:19:46.514 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-05-08T16:19:46.514 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-05-08T16:19:46.515 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-05-08T16:19:47.038 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-05-08T16:19:47.038 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-05-08T16:19:47.038 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-05-08T16:19:47.038 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:525: TEST_ec_backfill_unfound: ceph pg dump pgs 2024-05-08T16:19:47.427 INFO:tasks.workunit.client.0.smithi172.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-05-08T16:19:47.427 INFO:tasks.workunit.client.0.smithi172.stdout:2.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-05-08T16:19:41.120505+0000 0'0 44:18 [3,1,4,0,2] 3 [3,1,4,0,2] 3 0'0 2024-05-08T16:19:38.829889+0000 0'0 2024-05-08T16:19:38.829889+0000 0 0 periodic scrub scheduled @ 2024-05-10T03:11:05.455819+0000 0 0 2024-05-08T16:19:47.427 INFO:tasks.workunit.client.0.smithi172.stdout:1.3 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-05-08T16:19:41.227857+0000 0'0 43:12 [4,1,2] 4 [4,1,2] 4 0'0 2024-05-08T16:16:27.197992+0000 0'0 2024-05-08T16:16:27.197992+0000 0 0 periodic scrub scheduled @ 2024-05-10T02:10:13.781533+0000 0 0 2024-05-08T16:19:47.428 INFO:tasks.workunit.client.0.smithi172.stdout:1.2 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-05-08T16:19:41.077987+0000 0'0 44:97 [0,4,1] 0 [0,4,1] 0 0'0 2024-05-08T16:16:27.197992+0000 0'0 2024-05-08T16:16:27.197992+0000 0 0 periodic scrub scheduled @ 2024-05-09T19:42:51.689645+0000 0 0 2024-05-08T16:19:47.428 INFO:tasks.workunit.client.0.smithi172.stdout:1.1 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-05-08T16:19:41.227611+0000 0'0 43:12 [4,3,0] 4 [4,3,0] 4 0'0 2024-05-08T16:16:27.197992+0000 0'0 2024-05-08T16:16:27.197992+0000 0 0 periodic scrub scheduled @ 2024-05-10T02:18:12.944334+0000 0 0 2024-05-08T16:19:47.428 INFO:tasks.workunit.client.0.smithi172.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-05-08T16:18:21.668885+0000 0'0 44:67 [1,0,2] 1 [1,0,2] 1 0'0 2024-05-08T16:16:27.197992+0000 0'0 2024-05-08T16:16:27.197992+0000 0 0 periodic scrub scheduled @ 2024-05-09T16:42:43.448365+0000 0 0 2024-05-08T16:19:47.428 INFO:tasks.workunit.client.0.smithi172.stdout: 2024-05-08T16:19:47.428 INFO:tasks.workunit.client.0.smithi172.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-05-08T16:19:47.428 INFO:tasks.workunit.client.0.smithi172.stderr:dumped pgs 2024-05-08T16:19:47.445 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:527: TEST_ec_backfill_unfound: rados_put td/test-erasure-eio pool-jerasure myobject 2024-05-08T16:19:47.445 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:90: rados_put: local dir=td/test-erasure-eio 2024-05-08T16:19:47.446 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:91: rados_put: local poolname=pool-jerasure 2024-05-08T16:19:47.446 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:92: rados_put: local objname=myobject 2024-05-08T16:19:47.446 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-05-08T16:19:47.446 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 AAA 2024-05-08T16:19:47.446 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-05-08T16:19:47.446 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 BBB 2024-05-08T16:19:47.446 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-05-08T16:19:47.446 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 CCCC 2024-05-08T16:19:47.446 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-05-08T16:19:47.446 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 DDDD 2024-05-08T16:19:47.446 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:100: rados_put: rados --pool pool-jerasure put myobject td/test-erasure-eio/ORIGINAL 2024-05-08T16:19:47.626 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:528: TEST_ec_backfill_unfound: get_primary pool-jerasure myobject 2024-05-08T16:19:47.626 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=pool-jerasure 2024-05-08T16:19:47.626 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=myobject 2024-05-08T16:19:47.626 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map pool-jerasure myobject 2024-05-08T16:19:47.626 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-05-08T16:19:48.030 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:528: TEST_ec_backfill_unfound: local primary=3 2024-05-08T16:19:48.030 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:530: TEST_ec_backfill_unfound: get_osds pool-jerasure myobject 2024-05-08T16:19:48.031 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2024-05-08T16:19:48.031 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=myobject 2024-05-08T16:19:48.031 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure myobject 2024-05-08T16:19:48.031 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-05-08T16:19:48.451 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-05-08T16:19:48.451 INFO:tasks.workunit.client.0.smithi172.stderr:1 2024-05-08T16:19:48.451 INFO:tasks.workunit.client.0.smithi172.stderr:4 2024-05-08T16:19:48.451 INFO:tasks.workunit.client.0.smithi172.stderr:0 2024-05-08T16:19:48.451 INFO:tasks.workunit.client.0.smithi172.stderr:2' 2024-05-08T16:19:48.451 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 4 0 2 2024-05-08T16:19:48.451 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:530: TEST_ec_backfill_unfound: initial_osds=('3' '1' '4' '0' '2') 2024-05-08T16:19:48.451 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:530: TEST_ec_backfill_unfound: local -a initial_osds 2024-05-08T16:19:48.452 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:531: TEST_ec_backfill_unfound: local last_osd=2 2024-05-08T16:19:48.452 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:532: TEST_ec_backfill_unfound: kill_daemons td/test-erasure-eio TERM osd.2 2024-05-08T16:19:48.452 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-08T16:19:48.452 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-08T16:19:48.452 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-08T16:19:48.452 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-08T16:19:48.452 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-08T16:19:49.761 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-08T16:19:49.762 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:533: TEST_ec_backfill_unfound: ceph osd down 2 2024-05-08T16:19:50.328 INFO:tasks.workunit.client.0.smithi172.stderr:osd.2 is already down. 2024-05-08T16:19:50.346 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:534: TEST_ec_backfill_unfound: ceph osd out 2 2024-05-08T16:19:50.917 INFO:tasks.workunit.client.0.smithi172.stderr:osd.2 is already out. 2024-05-08T16:19:50.934 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:536: TEST_ec_backfill_unfound: ceph pg dump pgs 2024-05-08T16:19:51.463 INFO:tasks.workunit.client.0.smithi172.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-05-08T16:19:51.463 INFO:tasks.workunit.client.0.smithi172.stdout:2.0 1 0 0 0 0 4096 0 0 1 0 1 active+clean 2024-05-08T16:19:41.120505+0000 44'1 44:19 [3,1,4,0,2] 3 [3,1,4,0,2] 3 0'0 2024-05-08T16:19:38.829889+0000 0'0 2024-05-08T16:19:38.829889+0000 0 0 periodic scrub scheduled @ 2024-05-10T03:11:05.455819+0000 0 0 2024-05-08T16:19:51.463 INFO:tasks.workunit.client.0.smithi172.stdout:1.3 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-05-08T16:19:41.227857+0000 0'0 44:14 [4,1,2] 4 [4,1,2] 4 0'0 2024-05-08T16:16:27.197992+0000 0'0 2024-05-08T16:16:27.197992+0000 0 0 periodic scrub scheduled @ 2024-05-10T02:10:13.781533+0000 0 0 2024-05-08T16:19:51.463 INFO:tasks.workunit.client.0.smithi172.stdout:1.2 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-05-08T16:19:41.077987+0000 0'0 47:102 [0,4,1] 0 [0,4,1] 0 0'0 2024-05-08T16:16:27.197992+0000 0'0 2024-05-08T16:16:27.197992+0000 0 0 periodic scrub scheduled @ 2024-05-09T19:42:51.689645+0000 0 0 2024-05-08T16:19:51.463 INFO:tasks.workunit.client.0.smithi172.stdout:1.1 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-05-08T16:19:41.227611+0000 0'0 44:14 [4,3,0] 4 [4,3,0] 4 0'0 2024-05-08T16:16:27.197992+0000 0'0 2024-05-08T16:16:27.197992+0000 0 0 periodic scrub scheduled @ 2024-05-10T02:18:12.944334+0000 0 0 2024-05-08T16:19:51.464 INFO:tasks.workunit.client.0.smithi172.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-05-08T16:18:21.668885+0000 0'0 44:67 [1,0,2] 1 [1,0,2] 1 0'0 2024-05-08T16:16:27.197992+0000 0'0 2024-05-08T16:16:27.197992+0000 0 0 periodic scrub scheduled @ 2024-05-09T16:42:43.448365+0000 0 0 2024-05-08T16:19:51.464 INFO:tasks.workunit.client.0.smithi172.stdout: 2024-05-08T16:19:51.464 INFO:tasks.workunit.client.0.smithi172.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-05-08T16:19:51.464 INFO:tasks.workunit.client.0.smithi172.stderr:dumped pgs 2024-05-08T16:19:51.478 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:538: TEST_ec_backfill_unfound: dd if=/dev/urandom of=td/test-erasure-eio/ORIGINAL bs=1024 count=4 2024-05-08T16:19:51.480 INFO:tasks.workunit.client.0.smithi172.stderr:4+0 records in 2024-05-08T16:19:51.480 INFO:tasks.workunit.client.0.smithi172.stderr:4+0 records out 2024-05-08T16:19:51.480 INFO:tasks.workunit.client.0.smithi172.stderr:4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00029103 s, 14.1 MB/s 2024-05-08T16:19:51.480 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: seq 1 300 2024-05-08T16:19:51.481 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:19:51.481 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj1 td/test-erasure-eio/ORIGINAL 2024-05-08T16:19:51.952 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:19:51.952 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj2 td/test-erasure-eio/ORIGINAL 2024-05-08T16:19:52.500 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:19:52.500 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj3 td/test-erasure-eio/ORIGINAL 2024-05-08T16:19:52.802 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:19:52.802 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj4 td/test-erasure-eio/ORIGINAL 2024-05-08T16:19:53.112 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:19:53.112 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj5 td/test-erasure-eio/ORIGINAL 2024-05-08T16:19:53.405 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:19:53.405 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj6 td/test-erasure-eio/ORIGINAL 2024-05-08T16:19:53.666 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:19:53.667 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj7 td/test-erasure-eio/ORIGINAL 2024-05-08T16:19:53.951 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:19:53.952 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj8 td/test-erasure-eio/ORIGINAL 2024-05-08T16:19:54.240 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:19:54.240 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj9 td/test-erasure-eio/ORIGINAL 2024-05-08T16:19:54.548 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:19:54.549 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj10 td/test-erasure-eio/ORIGINAL 2024-05-08T16:19:54.839 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:19:54.840 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj11 td/test-erasure-eio/ORIGINAL 2024-05-08T16:19:55.117 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:19:55.117 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj12 td/test-erasure-eio/ORIGINAL 2024-05-08T16:19:55.386 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:19:55.386 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj13 td/test-erasure-eio/ORIGINAL 2024-05-08T16:19:55.679 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:19:55.679 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj14 td/test-erasure-eio/ORIGINAL 2024-05-08T16:19:56.032 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:19:56.032 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj15 td/test-erasure-eio/ORIGINAL 2024-05-08T16:19:56.318 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:19:56.318 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj16 td/test-erasure-eio/ORIGINAL 2024-05-08T16:19:56.619 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:19:56.619 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj17 td/test-erasure-eio/ORIGINAL 2024-05-08T16:19:56.870 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:19:56.870 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj18 td/test-erasure-eio/ORIGINAL 2024-05-08T16:19:57.180 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:19:57.180 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj19 td/test-erasure-eio/ORIGINAL 2024-05-08T16:19:57.474 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:19:57.475 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj20 td/test-erasure-eio/ORIGINAL 2024-05-08T16:19:57.742 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:19:57.742 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj21 td/test-erasure-eio/ORIGINAL 2024-05-08T16:19:58.035 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:19:58.036 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj22 td/test-erasure-eio/ORIGINAL 2024-05-08T16:19:58.346 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:19:58.346 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj23 td/test-erasure-eio/ORIGINAL 2024-05-08T16:19:58.656 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:19:58.656 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj24 td/test-erasure-eio/ORIGINAL 2024-05-08T16:19:58.941 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:19:58.941 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj25 td/test-erasure-eio/ORIGINAL 2024-05-08T16:19:59.184 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:19:59.184 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj26 td/test-erasure-eio/ORIGINAL 2024-05-08T16:19:59.444 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:19:59.444 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj27 td/test-erasure-eio/ORIGINAL 2024-05-08T16:19:59.754 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:19:59.754 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj28 td/test-erasure-eio/ORIGINAL 2024-05-08T16:20:00.048 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:20:00.049 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj29 td/test-erasure-eio/ORIGINAL 2024-05-08T16:20:00.342 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:20:00.342 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj30 td/test-erasure-eio/ORIGINAL 2024-05-08T16:20:00.609 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:20:00.610 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj31 td/test-erasure-eio/ORIGINAL 2024-05-08T16:20:00.886 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:20:00.887 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj32 td/test-erasure-eio/ORIGINAL 2024-05-08T16:20:01.172 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:20:01.172 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj33 td/test-erasure-eio/ORIGINAL 2024-05-08T16:20:01.474 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:20:01.474 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj34 td/test-erasure-eio/ORIGINAL 2024-05-08T16:20:01.726 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:20:01.726 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj35 td/test-erasure-eio/ORIGINAL 2024-05-08T16:20:02.010 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:20:02.010 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj36 td/test-erasure-eio/ORIGINAL 2024-05-08T16:20:02.321 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:20:02.321 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj37 td/test-erasure-eio/ORIGINAL 2024-05-08T16:20:02.631 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:20:02.632 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj38 td/test-erasure-eio/ORIGINAL 2024-05-08T16:20:02.924 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:20:02.924 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj39 td/test-erasure-eio/ORIGINAL 2024-05-08T16:20:03.176 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:20:03.176 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj40 td/test-erasure-eio/ORIGINAL 2024-05-08T16:20:03.428 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:20:03.428 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj41 td/test-erasure-eio/ORIGINAL 2024-05-08T16:20:03.767 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:20:03.767 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj42 td/test-erasure-eio/ORIGINAL 2024-05-08T16:20:04.090 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:20:04.090 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj43 td/test-erasure-eio/ORIGINAL 2024-05-08T16:20:04.409 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:20:04.409 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj44 td/test-erasure-eio/ORIGINAL 2024-05-08T16:20:04.831 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:20:04.831 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj45 td/test-erasure-eio/ORIGINAL 2024-05-08T16:20:05.130 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:20:05.130 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj46 td/test-erasure-eio/ORIGINAL 2024-05-08T16:20:05.464 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:20:05.464 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj47 td/test-erasure-eio/ORIGINAL 2024-05-08T16:20:05.784 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:20:05.784 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj48 td/test-erasure-eio/ORIGINAL 2024-05-08T16:20:06.034 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:20:06.034 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj49 td/test-erasure-eio/ORIGINAL 2024-05-08T16:20:06.336 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:20:06.336 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj50 td/test-erasure-eio/ORIGINAL 2024-05-08T16:20:06.664 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:20:06.664 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj51 td/test-erasure-eio/ORIGINAL 2024-05-08T16:20:06.958 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:20:06.958 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj52 td/test-erasure-eio/ORIGINAL 2024-05-08T16:20:07.259 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:20:07.259 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj53 td/test-erasure-eio/ORIGINAL 2024-05-08T16:20:07.502 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:20:07.502 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj54 td/test-erasure-eio/ORIGINAL 2024-05-08T16:20:07.813 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:20:07.813 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj55 td/test-erasure-eio/ORIGINAL 2024-05-08T16:20:08.164 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:20:08.165 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj56 td/test-erasure-eio/ORIGINAL 2024-05-08T16:20:08.475 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:20:08.476 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj57 td/test-erasure-eio/ORIGINAL 2024-05-08T16:20:08.727 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:20:08.728 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj58 td/test-erasure-eio/ORIGINAL 2024-05-08T16:20:08.979 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:20:08.979 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj59 td/test-erasure-eio/ORIGINAL 2024-05-08T16:20:09.289 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:20:09.289 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj60 td/test-erasure-eio/ORIGINAL 2024-05-08T16:20:09.640 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:20:09.641 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj61 td/test-erasure-eio/ORIGINAL 2024-05-08T16:20:09.926 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:20:09.926 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj62 td/test-erasure-eio/ORIGINAL 2024-05-08T16:20:10.202 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:20:10.202 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj63 td/test-erasure-eio/ORIGINAL 2024-05-08T16:20:10.513 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:20:10.513 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj64 td/test-erasure-eio/ORIGINAL 2024-05-08T16:20:10.831 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:20:10.831 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj65 td/test-erasure-eio/ORIGINAL 2024-05-08T16:20:11.157 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:20:11.158 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj66 td/test-erasure-eio/ORIGINAL 2024-05-08T16:20:11.411 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:20:11.411 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj67 td/test-erasure-eio/ORIGINAL 2024-05-08T16:20:11.636 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:20:11.636 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj68 td/test-erasure-eio/ORIGINAL 2024-05-08T16:20:11.914 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:20:11.914 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj69 td/test-erasure-eio/ORIGINAL 2024-05-08T16:20:12.265 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:20:12.265 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj70 td/test-erasure-eio/ORIGINAL 2024-05-08T16:20:12.584 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:20:12.584 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj71 td/test-erasure-eio/ORIGINAL 2024-05-08T16:20:12.843 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:20:12.843 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj72 td/test-erasure-eio/ORIGINAL 2024-05-08T16:20:13.096 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:20:13.096 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj73 td/test-erasure-eio/ORIGINAL 2024-05-08T16:20:13.414 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:20:13.414 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj74 td/test-erasure-eio/ORIGINAL 2024-05-08T16:20:13.758 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:20:13.759 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj75 td/test-erasure-eio/ORIGINAL 2024-05-08T16:20:14.043 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:20:14.043 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj76 td/test-erasure-eio/ORIGINAL 2024-05-08T16:20:14.345 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:20:14.345 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj77 td/test-erasure-eio/ORIGINAL 2024-05-08T16:20:14.699 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:20:14.699 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj78 td/test-erasure-eio/ORIGINAL 2024-05-08T16:20:15.123 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:20:15.124 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj79 td/test-erasure-eio/ORIGINAL 2024-05-08T16:20:15.426 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:20:15.426 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj80 td/test-erasure-eio/ORIGINAL 2024-05-08T16:20:15.736 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:20:15.736 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj81 td/test-erasure-eio/ORIGINAL 2024-05-08T16:20:16.046 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:20:16.046 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj82 td/test-erasure-eio/ORIGINAL 2024-05-08T16:20:16.373 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:20:16.374 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj83 td/test-erasure-eio/ORIGINAL 2024-05-08T16:20:16.667 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:20:16.667 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj84 td/test-erasure-eio/ORIGINAL 2024-05-08T16:20:16.927 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:20:16.927 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj85 td/test-erasure-eio/ORIGINAL 2024-05-08T16:20:17.206 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:20:17.206 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj86 td/test-erasure-eio/ORIGINAL 2024-05-08T16:20:17.457 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:20:17.458 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj87 td/test-erasure-eio/ORIGINAL 2024-05-08T16:20:17.717 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:20:17.718 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj88 td/test-erasure-eio/ORIGINAL 2024-05-08T16:20:18.003 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:20:18.003 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj89 td/test-erasure-eio/ORIGINAL 2024-05-08T16:20:18.346 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:20:18.346 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj90 td/test-erasure-eio/ORIGINAL 2024-05-08T16:20:18.674 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:20:18.674 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj91 td/test-erasure-eio/ORIGINAL 2024-05-08T16:20:18.959 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:20:18.959 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj92 td/test-erasure-eio/ORIGINAL 2024-05-08T16:20:19.229 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:20:19.229 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj93 td/test-erasure-eio/ORIGINAL 2024-05-08T16:20:19.545 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:20:19.545 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj94 td/test-erasure-eio/ORIGINAL 2024-05-08T16:20:19.873 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:20:19.873 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj95 td/test-erasure-eio/ORIGINAL 2024-05-08T16:20:20.141 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:20:20.141 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj96 td/test-erasure-eio/ORIGINAL 2024-05-08T16:20:20.418 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:20:20.418 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj97 td/test-erasure-eio/ORIGINAL 2024-05-08T16:20:20.686 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:20:20.686 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj98 td/test-erasure-eio/ORIGINAL 2024-05-08T16:20:20.963 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:20:20.963 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj99 td/test-erasure-eio/ORIGINAL 2024-05-08T16:20:21.272 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:20:21.272 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj100 td/test-erasure-eio/ORIGINAL 2024-05-08T16:20:21.532 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:20:21.532 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj101 td/test-erasure-eio/ORIGINAL 2024-05-08T16:20:21.834 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:20:21.834 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj102 td/test-erasure-eio/ORIGINAL 2024-05-08T16:20:22.161 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:20:22.161 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj103 td/test-erasure-eio/ORIGINAL 2024-05-08T16:20:22.480 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:20:22.480 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj104 td/test-erasure-eio/ORIGINAL 2024-05-08T16:20:22.791 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:20:22.791 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj105 td/test-erasure-eio/ORIGINAL 2024-05-08T16:20:23.050 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:20:23.050 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj106 td/test-erasure-eio/ORIGINAL 2024-05-08T16:20:23.336 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:20:23.336 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj107 td/test-erasure-eio/ORIGINAL 2024-05-08T16:20:23.611 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:20:23.611 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj108 td/test-erasure-eio/ORIGINAL 2024-05-08T16:20:23.947 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:20:23.947 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj109 td/test-erasure-eio/ORIGINAL 2024-05-08T16:20:24.258 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:20:24.258 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj110 td/test-erasure-eio/ORIGINAL 2024-05-08T16:20:24.561 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:20:24.561 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj111 td/test-erasure-eio/ORIGINAL 2024-05-08T16:20:24.828 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:20:24.828 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj112 td/test-erasure-eio/ORIGINAL 2024-05-08T16:20:25.229 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:20:25.230 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj113 td/test-erasure-eio/ORIGINAL 2024-05-08T16:20:25.581 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:20:25.582 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj114 td/test-erasure-eio/ORIGINAL 2024-05-08T16:20:25.883 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:20:25.883 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj115 td/test-erasure-eio/ORIGINAL 2024-05-08T16:20:26.128 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:20:26.129 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj116 td/test-erasure-eio/ORIGINAL 2024-05-08T16:20:26.430 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:20:26.430 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj117 td/test-erasure-eio/ORIGINAL 2024-05-08T16:20:26.755 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:20:26.755 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj118 td/test-erasure-eio/ORIGINAL 2024-05-08T16:20:27.058 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:20:27.058 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj119 td/test-erasure-eio/ORIGINAL 2024-05-08T16:20:27.292 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:20:27.292 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj120 td/test-erasure-eio/ORIGINAL 2024-05-08T16:20:27.537 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:20:27.537 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj121 td/test-erasure-eio/ORIGINAL 2024-05-08T16:20:27.863 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:20:27.863 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj122 td/test-erasure-eio/ORIGINAL 2024-05-08T16:20:28.214 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:20:28.215 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj123 td/test-erasure-eio/ORIGINAL 2024-05-08T16:20:28.519 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:20:28.519 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj124 td/test-erasure-eio/ORIGINAL 2024-05-08T16:20:28.760 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:20:28.760 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj125 td/test-erasure-eio/ORIGINAL 2024-05-08T16:20:29.071 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:20:29.071 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj126 td/test-erasure-eio/ORIGINAL 2024-05-08T16:20:29.389 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:20:29.389 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj127 td/test-erasure-eio/ORIGINAL 2024-05-08T16:20:29.740 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:20:29.740 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj128 td/test-erasure-eio/ORIGINAL 2024-05-08T16:20:30.033 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:20:30.033 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj129 td/test-erasure-eio/ORIGINAL 2024-05-08T16:20:30.319 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:20:30.319 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj130 td/test-erasure-eio/ORIGINAL 2024-05-08T16:20:30.637 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:20:30.637 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj131 td/test-erasure-eio/ORIGINAL 2024-05-08T16:20:30.972 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:20:30.972 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj132 td/test-erasure-eio/ORIGINAL 2024-05-08T16:20:31.274 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:20:31.274 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj133 td/test-erasure-eio/ORIGINAL 2024-05-08T16:20:31.518 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:20:31.518 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj134 td/test-erasure-eio/ORIGINAL 2024-05-08T16:20:31.812 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:20:31.812 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj135 td/test-erasure-eio/ORIGINAL 2024-05-08T16:20:32.137 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:20:32.137 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj136 td/test-erasure-eio/ORIGINAL 2024-05-08T16:20:32.482 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:20:32.482 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj137 td/test-erasure-eio/ORIGINAL 2024-05-08T16:20:32.766 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:20:32.766 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj138 td/test-erasure-eio/ORIGINAL 2024-05-08T16:20:33.009 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:20:33.010 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj139 td/test-erasure-eio/ORIGINAL 2024-05-08T16:20:33.278 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:20:33.278 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj140 td/test-erasure-eio/ORIGINAL 2024-05-08T16:20:33.604 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:20:33.604 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj141 td/test-erasure-eio/ORIGINAL 2024-05-08T16:20:33.973 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:20:33.973 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj142 td/test-erasure-eio/ORIGINAL 2024-05-08T16:20:34.284 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:20:34.284 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj143 td/test-erasure-eio/ORIGINAL 2024-05-08T16:20:34.553 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:20:34.553 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj144 td/test-erasure-eio/ORIGINAL 2024-05-08T16:20:34.870 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:20:34.871 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj145 td/test-erasure-eio/ORIGINAL 2024-05-08T16:20:35.206 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:20:35.207 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj146 td/test-erasure-eio/ORIGINAL 2024-05-08T16:20:35.750 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:20:35.750 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj147 td/test-erasure-eio/ORIGINAL 2024-05-08T16:20:36.051 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:20:36.051 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj148 td/test-erasure-eio/ORIGINAL 2024-05-08T16:20:36.345 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:20:36.345 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj149 td/test-erasure-eio/ORIGINAL 2024-05-08T16:20:36.697 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:20:36.697 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj150 td/test-erasure-eio/ORIGINAL 2024-05-08T16:20:37.006 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:20:37.007 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj151 td/test-erasure-eio/ORIGINAL 2024-05-08T16:20:37.301 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:20:37.301 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj152 td/test-erasure-eio/ORIGINAL 2024-05-08T16:20:37.544 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:20:37.544 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj153 td/test-erasure-eio/ORIGINAL 2024-05-08T16:20:37.878 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:20:37.878 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj154 td/test-erasure-eio/ORIGINAL 2024-05-08T16:20:38.230 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:20:38.231 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj155 td/test-erasure-eio/ORIGINAL 2024-05-08T16:20:38.551 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:20:38.551 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj156 td/test-erasure-eio/ORIGINAL 2024-05-08T16:20:38.826 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:20:38.826 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj157 td/test-erasure-eio/ORIGINAL 2024-05-08T16:20:39.111 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:20:39.111 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj158 td/test-erasure-eio/ORIGINAL 2024-05-08T16:20:39.429 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:20:39.430 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj159 td/test-erasure-eio/ORIGINAL 2024-05-08T16:20:39.815 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:20:39.815 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj160 td/test-erasure-eio/ORIGINAL 2024-05-08T16:20:40.101 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:20:40.101 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj161 td/test-erasure-eio/ORIGINAL 2024-05-08T16:20:40.392 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:20:40.393 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj162 td/test-erasure-eio/ORIGINAL 2024-05-08T16:20:40.705 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:20:40.706 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj163 td/test-erasure-eio/ORIGINAL 2024-05-08T16:20:41.040 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:20:41.040 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj164 td/test-erasure-eio/ORIGINAL 2024-05-08T16:20:41.357 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:20:41.357 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj165 td/test-erasure-eio/ORIGINAL 2024-05-08T16:20:41.617 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:20:41.617 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj166 td/test-erasure-eio/ORIGINAL 2024-05-08T16:20:41.894 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:20:41.895 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj167 td/test-erasure-eio/ORIGINAL 2024-05-08T16:20:42.188 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:20:42.188 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj168 td/test-erasure-eio/ORIGINAL 2024-05-08T16:20:42.556 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:20:42.556 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj169 td/test-erasure-eio/ORIGINAL 2024-05-08T16:20:42.850 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:20:42.850 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj170 td/test-erasure-eio/ORIGINAL 2024-05-08T16:20:43.109 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:20:43.109 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj171 td/test-erasure-eio/ORIGINAL 2024-05-08T16:20:43.436 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:20:43.437 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj172 td/test-erasure-eio/ORIGINAL 2024-05-08T16:20:43.788 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:20:43.788 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj173 td/test-erasure-eio/ORIGINAL 2024-05-08T16:20:44.142 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:20:44.142 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj174 td/test-erasure-eio/ORIGINAL 2024-05-08T16:20:44.442 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:20:44.442 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj175 td/test-erasure-eio/ORIGINAL 2024-05-08T16:20:44.719 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:20:44.719 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj176 td/test-erasure-eio/ORIGINAL 2024-05-08T16:20:45.029 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:20:45.029 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj177 td/test-erasure-eio/ORIGINAL 2024-05-08T16:20:45.381 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:20:45.381 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj178 td/test-erasure-eio/ORIGINAL 2024-05-08T16:20:45.683 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:20:45.684 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj179 td/test-erasure-eio/ORIGINAL 2024-05-08T16:20:46.044 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:20:46.044 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj180 td/test-erasure-eio/ORIGINAL 2024-05-08T16:20:46.337 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:20:46.337 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj181 td/test-erasure-eio/ORIGINAL 2024-05-08T16:20:46.663 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:20:46.663 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj182 td/test-erasure-eio/ORIGINAL 2024-05-08T16:20:47.015 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:20:47.015 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj183 td/test-erasure-eio/ORIGINAL 2024-05-08T16:20:47.284 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:20:47.284 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj184 td/test-erasure-eio/ORIGINAL 2024-05-08T16:20:47.552 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:20:47.552 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj185 td/test-erasure-eio/ORIGINAL 2024-05-08T16:20:47.845 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:20:47.846 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj186 td/test-erasure-eio/ORIGINAL 2024-05-08T16:20:48.156 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:20:48.156 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj187 td/test-erasure-eio/ORIGINAL 2024-05-08T16:20:48.483 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:20:48.483 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj188 td/test-erasure-eio/ORIGINAL 2024-05-08T16:20:48.758 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:20:48.758 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj189 td/test-erasure-eio/ORIGINAL 2024-05-08T16:20:49.047 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:20:49.047 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj190 td/test-erasure-eio/ORIGINAL 2024-05-08T16:20:49.362 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:20:49.362 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj191 td/test-erasure-eio/ORIGINAL 2024-05-08T16:20:49.698 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:20:49.698 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj192 td/test-erasure-eio/ORIGINAL 2024-05-08T16:20:49.983 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:20:49.983 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj193 td/test-erasure-eio/ORIGINAL 2024-05-08T16:20:50.295 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:20:50.295 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj194 td/test-erasure-eio/ORIGINAL 2024-05-08T16:20:50.595 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:20:50.595 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj195 td/test-erasure-eio/ORIGINAL 2024-05-08T16:20:50.888 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:20:50.888 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj196 td/test-erasure-eio/ORIGINAL 2024-05-08T16:20:51.198 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:20:51.198 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj197 td/test-erasure-eio/ORIGINAL 2024-05-08T16:20:51.483 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:20:51.483 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj198 td/test-erasure-eio/ORIGINAL 2024-05-08T16:20:51.786 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:20:51.786 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj199 td/test-erasure-eio/ORIGINAL 2024-05-08T16:20:52.088 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:20:52.089 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj200 td/test-erasure-eio/ORIGINAL 2024-05-08T16:20:52.431 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:20:52.431 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj201 td/test-erasure-eio/ORIGINAL 2024-05-08T16:20:52.758 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:20:52.758 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj202 td/test-erasure-eio/ORIGINAL 2024-05-08T16:20:53.035 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:20:53.035 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj203 td/test-erasure-eio/ORIGINAL 2024-05-08T16:20:53.319 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:20:53.319 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj204 td/test-erasure-eio/ORIGINAL 2024-05-08T16:20:53.604 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:20:53.604 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj205 td/test-erasure-eio/ORIGINAL 2024-05-08T16:20:53.990 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:20:53.990 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj206 td/test-erasure-eio/ORIGINAL 2024-05-08T16:20:54.300 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:20:54.301 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj207 td/test-erasure-eio/ORIGINAL 2024-05-08T16:20:54.602 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:20:54.602 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj208 td/test-erasure-eio/ORIGINAL 2024-05-08T16:20:54.920 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:20:54.920 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj209 td/test-erasure-eio/ORIGINAL 2024-05-08T16:20:55.238 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:20:55.238 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj210 td/test-erasure-eio/ORIGINAL 2024-05-08T16:20:55.566 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:20:55.566 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj211 td/test-erasure-eio/ORIGINAL 2024-05-08T16:20:55.851 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:20:55.851 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj212 td/test-erasure-eio/ORIGINAL 2024-05-08T16:20:56.244 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:20:56.245 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj213 td/test-erasure-eio/ORIGINAL 2024-05-08T16:20:56.572 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:20:56.572 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj214 td/test-erasure-eio/ORIGINAL 2024-05-08T16:20:56.856 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:20:56.856 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj215 td/test-erasure-eio/ORIGINAL 2024-05-08T16:20:57.134 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:20:57.134 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj216 td/test-erasure-eio/ORIGINAL 2024-05-08T16:20:57.428 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:20:57.428 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj217 td/test-erasure-eio/ORIGINAL 2024-05-08T16:20:57.796 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:20:57.796 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj218 td/test-erasure-eio/ORIGINAL 2024-05-08T16:20:58.106 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:20:58.106 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj219 td/test-erasure-eio/ORIGINAL 2024-05-08T16:20:58.393 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:20:58.393 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj220 td/test-erasure-eio/ORIGINAL 2024-05-08T16:20:58.668 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:20:58.669 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj221 td/test-erasure-eio/ORIGINAL 2024-05-08T16:20:59.011 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:20:59.011 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj222 td/test-erasure-eio/ORIGINAL 2024-05-08T16:20:59.357 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:20:59.357 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj223 td/test-erasure-eio/ORIGINAL 2024-05-08T16:20:59.617 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:20:59.617 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj224 td/test-erasure-eio/ORIGINAL 2024-05-08T16:20:59.893 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:20:59.893 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj225 td/test-erasure-eio/ORIGINAL 2024-05-08T16:21:00.220 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:21:00.220 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj226 td/test-erasure-eio/ORIGINAL 2024-05-08T16:21:00.547 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:21:00.547 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj227 td/test-erasure-eio/ORIGINAL 2024-05-08T16:21:00.849 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:21:00.849 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj228 td/test-erasure-eio/ORIGINAL 2024-05-08T16:21:01.159 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:21:01.159 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj229 td/test-erasure-eio/ORIGINAL 2024-05-08T16:21:01.472 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:21:01.472 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj230 td/test-erasure-eio/ORIGINAL 2024-05-08T16:21:01.811 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:21:01.812 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj231 td/test-erasure-eio/ORIGINAL 2024-05-08T16:21:02.113 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:21:02.113 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj232 td/test-erasure-eio/ORIGINAL 2024-05-08T16:21:02.416 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:21:02.416 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj233 td/test-erasure-eio/ORIGINAL 2024-05-08T16:21:02.675 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:21:02.675 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj234 td/test-erasure-eio/ORIGINAL 2024-05-08T16:21:02.993 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:21:02.993 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj235 td/test-erasure-eio/ORIGINAL 2024-05-08T16:21:03.354 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:21:03.354 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj236 td/test-erasure-eio/ORIGINAL 2024-05-08T16:21:03.665 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:21:03.665 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj237 td/test-erasure-eio/ORIGINAL 2024-05-08T16:21:03.949 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:21:03.949 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj238 td/test-erasure-eio/ORIGINAL 2024-05-08T16:21:04.259 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:21:04.259 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj239 td/test-erasure-eio/ORIGINAL 2024-05-08T16:21:04.610 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:21:04.610 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj240 td/test-erasure-eio/ORIGINAL 2024-05-08T16:21:04.954 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:21:04.955 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj241 td/test-erasure-eio/ORIGINAL 2024-05-08T16:21:05.239 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:21:05.239 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj242 td/test-erasure-eio/ORIGINAL 2024-05-08T16:21:05.491 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:21:05.491 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj243 td/test-erasure-eio/ORIGINAL 2024-05-08T16:21:05.793 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:21:05.793 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj244 td/test-erasure-eio/ORIGINAL 2024-05-08T16:21:06.111 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:21:06.112 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj245 td/test-erasure-eio/ORIGINAL 2024-05-08T16:21:06.697 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:21:06.697 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj246 td/test-erasure-eio/ORIGINAL 2024-05-08T16:21:06.972 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:21:06.972 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj247 td/test-erasure-eio/ORIGINAL 2024-05-08T16:21:07.242 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:21:07.242 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj248 td/test-erasure-eio/ORIGINAL 2024-05-08T16:21:07.544 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:21:07.544 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj249 td/test-erasure-eio/ORIGINAL 2024-05-08T16:21:07.879 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:21:07.879 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj250 td/test-erasure-eio/ORIGINAL 2024-05-08T16:21:08.172 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:21:08.172 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj251 td/test-erasure-eio/ORIGINAL 2024-05-08T16:21:08.482 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:21:08.482 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj252 td/test-erasure-eio/ORIGINAL 2024-05-08T16:21:08.766 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:21:08.767 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj253 td/test-erasure-eio/ORIGINAL 2024-05-08T16:21:09.070 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:21:09.070 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj254 td/test-erasure-eio/ORIGINAL 2024-05-08T16:21:09.403 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:21:09.403 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj255 td/test-erasure-eio/ORIGINAL 2024-05-08T16:21:09.721 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:21:09.722 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj256 td/test-erasure-eio/ORIGINAL 2024-05-08T16:21:09.998 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:21:09.998 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj257 td/test-erasure-eio/ORIGINAL 2024-05-08T16:21:10.284 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:21:10.284 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj258 td/test-erasure-eio/ORIGINAL 2024-05-08T16:21:10.594 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:21:10.595 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj259 td/test-erasure-eio/ORIGINAL 2024-05-08T16:21:10.937 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:21:10.937 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj260 td/test-erasure-eio/ORIGINAL 2024-05-08T16:21:11.198 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:21:11.198 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj261 td/test-erasure-eio/ORIGINAL 2024-05-08T16:21:11.482 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:21:11.482 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj262 td/test-erasure-eio/ORIGINAL 2024-05-08T16:21:11.792 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:21:11.793 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj263 td/test-erasure-eio/ORIGINAL 2024-05-08T16:21:12.138 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:21:12.138 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj264 td/test-erasure-eio/ORIGINAL 2024-05-08T16:21:12.471 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:21:12.471 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj265 td/test-erasure-eio/ORIGINAL 2024-05-08T16:21:12.739 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:21:12.739 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj266 td/test-erasure-eio/ORIGINAL 2024-05-08T16:21:13.007 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:21:13.007 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj267 td/test-erasure-eio/ORIGINAL 2024-05-08T16:21:13.317 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:21:13.317 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj268 td/test-erasure-eio/ORIGINAL 2024-05-08T16:21:13.636 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:21:13.636 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj269 td/test-erasure-eio/ORIGINAL 2024-05-08T16:21:13.955 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:21:13.955 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj270 td/test-erasure-eio/ORIGINAL 2024-05-08T16:21:14.231 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:21:14.232 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj271 td/test-erasure-eio/ORIGINAL 2024-05-08T16:21:14.533 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:21:14.533 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj272 td/test-erasure-eio/ORIGINAL 2024-05-08T16:21:14.860 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:21:14.860 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj273 td/test-erasure-eio/ORIGINAL 2024-05-08T16:21:15.196 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:21:15.196 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj274 td/test-erasure-eio/ORIGINAL 2024-05-08T16:21:15.562 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:21:15.562 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj275 td/test-erasure-eio/ORIGINAL 2024-05-08T16:21:15.865 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:21:15.865 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj276 td/test-erasure-eio/ORIGINAL 2024-05-08T16:21:16.150 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:21:16.150 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj277 td/test-erasure-eio/ORIGINAL 2024-05-08T16:21:16.527 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:21:16.527 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj278 td/test-erasure-eio/ORIGINAL 2024-05-08T16:21:16.971 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:21:16.971 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj279 td/test-erasure-eio/ORIGINAL 2024-05-08T16:21:17.238 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:21:17.238 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj280 td/test-erasure-eio/ORIGINAL 2024-05-08T16:21:17.507 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:21:17.507 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj281 td/test-erasure-eio/ORIGINAL 2024-05-08T16:21:17.808 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:21:17.809 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj282 td/test-erasure-eio/ORIGINAL 2024-05-08T16:21:18.152 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:21:18.152 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj283 td/test-erasure-eio/ORIGINAL 2024-05-08T16:21:18.482 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:21:18.482 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj284 td/test-erasure-eio/ORIGINAL 2024-05-08T16:21:18.739 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:21:18.739 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj285 td/test-erasure-eio/ORIGINAL 2024-05-08T16:21:19.082 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:21:19.082 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj286 td/test-erasure-eio/ORIGINAL 2024-05-08T16:21:19.401 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:21:19.402 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj287 td/test-erasure-eio/ORIGINAL 2024-05-08T16:21:19.744 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:21:19.744 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj288 td/test-erasure-eio/ORIGINAL 2024-05-08T16:21:20.037 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:21:20.037 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj289 td/test-erasure-eio/ORIGINAL 2024-05-08T16:21:20.331 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:21:20.332 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj290 td/test-erasure-eio/ORIGINAL 2024-05-08T16:21:20.642 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:21:20.642 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj291 td/test-erasure-eio/ORIGINAL 2024-05-08T16:21:20.947 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:21:20.947 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj292 td/test-erasure-eio/ORIGINAL 2024-05-08T16:21:21.252 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:21:21.252 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj293 td/test-erasure-eio/ORIGINAL 2024-05-08T16:21:21.528 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:21:21.528 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj294 td/test-erasure-eio/ORIGINAL 2024-05-08T16:21:21.839 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:21:21.839 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj295 td/test-erasure-eio/ORIGINAL 2024-05-08T16:21:22.108 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:21:22.108 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj296 td/test-erasure-eio/ORIGINAL 2024-05-08T16:21:22.418 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:21:22.419 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj297 td/test-erasure-eio/ORIGINAL 2024-05-08T16:21:22.762 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:21:22.762 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj298 td/test-erasure-eio/ORIGINAL 2024-05-08T16:21:23.046 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:21:23.047 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj299 td/test-erasure-eio/ORIGINAL 2024-05-08T16:21:23.347 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:539: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:21:23.347 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:541: TEST_ec_backfill_unfound: rados --pool pool-jerasure put obj300 td/test-erasure-eio/ORIGINAL 2024-05-08T16:21:23.635 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:544: TEST_ec_backfill_unfound: inject_eio ec data pool-jerasure obj250 td/test-erasure-eio 0 2024-05-08T16:21:23.635 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2394: inject_eio: local pooltype=ec 2024-05-08T16:21:23.635 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2395: inject_eio: shift 2024-05-08T16:21:23.635 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2396: inject_eio: local which=data 2024-05-08T16:21:23.635 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2397: inject_eio: shift 2024-05-08T16:21:23.635 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2398: inject_eio: local poolname=pool-jerasure 2024-05-08T16:21:23.635 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2399: inject_eio: shift 2024-05-08T16:21:23.635 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2400: inject_eio: local objname=obj250 2024-05-08T16:21:23.635 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2401: inject_eio: shift 2024-05-08T16:21:23.636 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2402: inject_eio: local dir=td/test-erasure-eio 2024-05-08T16:21:23.636 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2403: inject_eio: shift 2024-05-08T16:21:23.636 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2404: inject_eio: local shard_id=0 2024-05-08T16:21:23.636 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2405: inject_eio: shift 2024-05-08T16:21:23.636 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: get_osds pool-jerasure obj250 2024-05-08T16:21:23.636 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2024-05-08T16:21:23.636 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj250 2024-05-08T16:21:23.636 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure obj250 2024-05-08T16:21:23.636 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-05-08T16:21:24.061 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-05-08T16:21:24.062 INFO:tasks.workunit.client.0.smithi172.stderr:1 2024-05-08T16:21:24.062 INFO:tasks.workunit.client.0.smithi172.stderr:4 2024-05-08T16:21:24.062 INFO:tasks.workunit.client.0.smithi172.stderr:0 2024-05-08T16:21:24.062 INFO:tasks.workunit.client.0.smithi172.stderr:2147483647' 2024-05-08T16:21:24.062 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 4 0 2147483647 2024-05-08T16:21:24.062 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: initial_osds=('3' '1' '4' '0' '2147483647') 2024-05-08T16:21:24.062 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: local -a initial_osds 2024-05-08T16:21:24.062 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2408: inject_eio: local osd_id=3 2024-05-08T16:21:24.062 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2409: inject_eio: '[' ec '!=' ec ']' 2024-05-08T16:21:24.062 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: cat td/test-erasure-eio/3/type 2024-05-08T16:21:24.064 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: type=bluestore 2024-05-08T16:21:24.064 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: set_config osd 3 bluestore_debug_inject_read_err true 2024-05-08T16:21:24.064 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-05-08T16:21:24.065 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-05-08T16:21:24.065 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-05-08T16:21:24.065 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-05-08T16:21:24.065 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-05-08T16:21:24.065 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-05-08T16:21:24.065 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-05-08T16:21:24.065 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-05-08T16:21:24.065 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-08T16:21:24.065 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T16:21:24.065 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81110 2024-05-08T16:21:24.065 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.81110/ceph-osd.3.asok 2024-05-08T16:21:24.065 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.81110/ceph-osd.3.asok config set bluestore_debug_inject_read_err true 2024-05-08T16:21:24.178 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-05-08T16:21:24.179 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local loop=0 2024-05-08T16:21:24.179 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: grep -q Invalid 2024-05-08T16:21:24.180 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: get_asok_path osd.3 2024-05-08T16:21:24.180 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-05-08T16:21:24.180 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-05-08T16:21:24.181 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-08T16:21:24.181 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T16:21:24.181 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81110 2024-05-08T16:21:24.181 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.81110/ceph-osd.3.asok 2024-05-08T16:21:24.181 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: CEPH_ARGS= 2024-05-08T16:21:24.181 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: ceph --admin-daemon /tmp/ceph-asok.81110/ceph-osd.3.asok injectdataerr pool-jerasure obj250 0 2024-05-08T16:21:24.305 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:545: TEST_ec_backfill_unfound: inject_eio ec data pool-jerasure obj250 td/test-erasure-eio 1 2024-05-08T16:21:24.306 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2394: inject_eio: local pooltype=ec 2024-05-08T16:21:24.306 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2395: inject_eio: shift 2024-05-08T16:21:24.306 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2396: inject_eio: local which=data 2024-05-08T16:21:24.306 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2397: inject_eio: shift 2024-05-08T16:21:24.306 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2398: inject_eio: local poolname=pool-jerasure 2024-05-08T16:21:24.306 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2399: inject_eio: shift 2024-05-08T16:21:24.306 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2400: inject_eio: local objname=obj250 2024-05-08T16:21:24.306 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2401: inject_eio: shift 2024-05-08T16:21:24.306 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2402: inject_eio: local dir=td/test-erasure-eio 2024-05-08T16:21:24.306 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2403: inject_eio: shift 2024-05-08T16:21:24.306 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2404: inject_eio: local shard_id=1 2024-05-08T16:21:24.306 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2405: inject_eio: shift 2024-05-08T16:21:24.307 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: get_osds pool-jerasure obj250 2024-05-08T16:21:24.307 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2024-05-08T16:21:24.307 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj250 2024-05-08T16:21:24.307 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure obj250 2024-05-08T16:21:24.307 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-05-08T16:21:24.708 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-05-08T16:21:24.709 INFO:tasks.workunit.client.0.smithi172.stderr:1 2024-05-08T16:21:24.709 INFO:tasks.workunit.client.0.smithi172.stderr:4 2024-05-08T16:21:24.709 INFO:tasks.workunit.client.0.smithi172.stderr:0 2024-05-08T16:21:24.709 INFO:tasks.workunit.client.0.smithi172.stderr:2147483647' 2024-05-08T16:21:24.709 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 4 0 2147483647 2024-05-08T16:21:24.709 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: initial_osds=('3' '1' '4' '0' '2147483647') 2024-05-08T16:21:24.709 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: local -a initial_osds 2024-05-08T16:21:24.709 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2408: inject_eio: local osd_id=1 2024-05-08T16:21:24.709 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2409: inject_eio: '[' ec '!=' ec ']' 2024-05-08T16:21:24.709 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: cat td/test-erasure-eio/1/type 2024-05-08T16:21:24.711 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: type=bluestore 2024-05-08T16:21:24.711 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2024-05-08T16:21:24.711 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-05-08T16:21:24.711 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=1 2024-05-08T16:21:24.712 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-05-08T16:21:24.712 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-05-08T16:21:24.712 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-05-08T16:21:24.712 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.1 2024-05-08T16:21:24.712 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-05-08T16:21:24.712 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-05-08T16:21:24.712 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-08T16:21:24.712 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T16:21:24.712 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81110 2024-05-08T16:21:24.712 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.81110/ceph-osd.1.asok 2024-05-08T16:21:24.712 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.81110/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2024-05-08T16:21:24.817 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-05-08T16:21:24.818 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local loop=0 2024-05-08T16:21:24.818 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: grep -q Invalid 2024-05-08T16:21:24.819 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: get_asok_path osd.1 2024-05-08T16:21:24.819 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-05-08T16:21:24.819 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-05-08T16:21:24.820 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-08T16:21:24.820 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T16:21:24.820 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81110 2024-05-08T16:21:24.820 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.81110/ceph-osd.1.asok 2024-05-08T16:21:24.820 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: CEPH_ARGS= 2024-05-08T16:21:24.820 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: ceph --admin-daemon /tmp/ceph-asok.81110/ceph-osd.1.asok injectdataerr pool-jerasure obj250 1 2024-05-08T16:21:24.932 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:547: TEST_ec_backfill_unfound: activate_osd td/test-erasure-eio 2 2024-05-08T16:21:24.932 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/test-erasure-eio 2024-05-08T16:21:24.932 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-05-08T16:21:24.932 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-05-08T16:21:24.932 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-05-08T16:21:24.932 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/test-erasure-eio/2 2024-05-08T16:21:24.933 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=4457b0f1-028b-482b-8ee3-7ee05861e6cc --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-05-08T16:21:24.933 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-08T16:21:24.933 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-05-08T16:21:24.933 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-08T16:21:24.933 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/2' 2024-05-08T16:21:24.933 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/2/journal' 2024-05-08T16:21:24.933 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-05-08T16:21:24.933 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-05-08T16:21:24.933 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-05-08T16:21:24.933 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-05-08T16:21:24.933 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-08T16:21:24.934 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-08T16:21:24.934 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-08T16:21:24.934 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T16:21:24.934 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81110 2024-05-08T16:21:24.934 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.81110/$cluster-$name.asok' 2024-05-08T16:21:24.934 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.81110/$cluster-$name.asok' 2024-05-08T16:21:24.934 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-05-08T16:21:24.934 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-05-08T16:21:24.934 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-05-08T16:21:24.934 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-08T16:21:24.934 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-08T16:21:24.935 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-08T16:21:24.935 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-08T16:21:24.935 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-05-08T16:21:24.935 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-05-08T16:21:24.935 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/test-erasure-eio/2 2024-05-08T16:21:24.935 INFO:tasks.workunit.client.0.smithi172.stdout:start osd.2 2024-05-08T16:21:24.936 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-05-08T16:21:24.936 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=4457b0f1-028b-482b-8ee3-7ee05861e6cc --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81110/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-05-08T16:21:24.937 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/test-erasure-eio/2/whoami 2024-05-08T16:21:24.937 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-05-08T16:21:24.937 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-05-08T16:21:24.937 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-05-08T16:21:24.937 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-05-08T16:21:24.962 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:21:24.966+0000 7f0e6379f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:21:24.964 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:21:24.970+0000 7f0e6379f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:21:24.965 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:21:24.970+0000 7f0e6379f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:21:25.332 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-05-08T16:21:25.332 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-08T16:21:25.332 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-05-08T16:21:25.332 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-08T16:21:25.333 INFO:tasks.workunit.client.0.smithi172.stdout:0 2024-05-08T16:21:25.333 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-08T16:21:25.333 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:21:25.333 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-08T16:21:25.333 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:21:25.333 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T16:21:25.512 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:21:25.517+0000 7f0e6379f7c0 -1 Falling back to public interface 2024-05-08T16:21:25.726 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:21:26.728 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:21:26.728 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:21:26.728 INFO:tasks.workunit.client.0.smithi172.stdout:1 2024-05-08T16:21:26.729 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-08T16:21:26.729 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:21:26.729 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T16:21:27.129 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:21:28.130 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:21:28.130 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:21:28.131 INFO:tasks.workunit.client.0.smithi172.stdout:2 2024-05-08T16:21:28.131 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-08T16:21:28.131 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:21:28.131 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T16:21:28.618 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:21:28.758 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:21:28.761+0000 7f0e6379f7c0 -1 osd.2 44 log_to_monitors true 2024-05-08T16:21:29.619 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:21:29.619 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:21:29.620 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-05-08T16:21:29.620 INFO:tasks.workunit.client.0.smithi172.stdout:3 2024-05-08T16:21:29.620 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:21:29.620 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T16:21:30.180 INFO:tasks.workunit.client.0.smithi172.stdout:osd.2 up out weight 0 up_from 52 up_thru 20 down_at 45 last_clean_interval [18,44) [v2:127.0.0.1:6818/3073519814,v1:127.0.0.1:6819/3073519814] [v2:127.0.0.1:6820/3073519814,v1:127.0.0.1:6821/3073519814] exists,up 91b6118c-0348-4a21-beef-0db38b0f7cce 2024-05-08T16:21:30.181 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-08T16:21:30.181 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-08T16:21:30.181 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-08T16:21:30.181 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:548: TEST_ec_backfill_unfound: ceph osd in 2 2024-05-08T16:21:31.897 INFO:tasks.workunit.client.0.smithi172.stderr:osd.2 is already in. 2024-05-08T16:21:31.917 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:550: TEST_ec_backfill_unfound: sleep 15 2024-05-08T16:21:46.920 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:552: TEST_ec_backfill_unfound: seq 1 240 2024-05-08T16:21:46.921 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:552: TEST_ec_backfill_unfound: for tmp in $(seq 1 240) 2024-05-08T16:21:46.927 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:553: TEST_ec_backfill_unfound: get_state 2.0 2024-05-08T16:21:46.927 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:60: get_state: local pgid=2.0 2024-05-08T16:21:46.927 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:61: get_state: local sname=state 2024-05-08T16:21:46.928 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:62: get_state: ceph --format json pg dump pgs 2024-05-08T16:21:46.928 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:63: get_state: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .state' 2024-05-08T16:21:47.337 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:553: TEST_ec_backfill_unfound: state=active+undersized+degraded+remapped+backfilling 2024-05-08T16:21:47.337 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:554: TEST_ec_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2024-05-08T16:21:47.337 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:554: TEST_ec_backfill_unfound: grep backfill_unfound 2024-05-08T16:21:47.339 INFO:tasks.workunit.client.0.smithi172.stdout:active+undersized+degraded+remapped+backfilling 2024-05-08T16:21:47.340 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:555: TEST_ec_backfill_unfound: '[' 1 = 0 ']' 2024-05-08T16:21:47.340 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:558: TEST_ec_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2024-05-08T16:21:47.340 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: sleep 1 2024-05-08T16:21:48.340 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:552: TEST_ec_backfill_unfound: for tmp in $(seq 1 240) 2024-05-08T16:21:48.342 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:553: TEST_ec_backfill_unfound: get_state 2.0 2024-05-08T16:21:48.342 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:60: get_state: local pgid=2.0 2024-05-08T16:21:48.342 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:61: get_state: local sname=state 2024-05-08T16:21:48.342 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:62: get_state: ceph --format json pg dump pgs 2024-05-08T16:21:48.342 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:63: get_state: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .state' 2024-05-08T16:21:48.758 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:553: TEST_ec_backfill_unfound: state=active+undersized+degraded+remapped+backfilling 2024-05-08T16:21:48.759 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:554: TEST_ec_backfill_unfound: grep backfill_unfound 2024-05-08T16:21:48.759 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:554: TEST_ec_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2024-05-08T16:21:48.759 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:555: TEST_ec_backfill_unfound: '[' 1 = 0 ']' 2024-05-08T16:21:48.760 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:558: TEST_ec_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2024-05-08T16:21:48.760 INFO:tasks.workunit.client.0.smithi172.stdout:active+undersized+degraded+remapped+backfilling 2024-05-08T16:21:48.760 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: sleep 1 2024-05-08T16:21:49.761 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:552: TEST_ec_backfill_unfound: for tmp in $(seq 1 240) 2024-05-08T16:21:49.762 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:553: TEST_ec_backfill_unfound: get_state 2.0 2024-05-08T16:21:49.762 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:60: get_state: local pgid=2.0 2024-05-08T16:21:49.762 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:61: get_state: local sname=state 2024-05-08T16:21:49.763 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:62: get_state: ceph --format json pg dump pgs 2024-05-08T16:21:49.763 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:63: get_state: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .state' 2024-05-08T16:21:50.180 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:553: TEST_ec_backfill_unfound: state=active+undersized+degraded+remapped+backfilling 2024-05-08T16:21:50.181 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:554: TEST_ec_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2024-05-08T16:21:50.181 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:554: TEST_ec_backfill_unfound: grep backfill_unfound 2024-05-08T16:21:50.182 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:555: TEST_ec_backfill_unfound: '[' 1 = 0 ']' 2024-05-08T16:21:50.182 INFO:tasks.workunit.client.0.smithi172.stdout:active+undersized+degraded+remapped+backfilling 2024-05-08T16:21:50.183 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:558: TEST_ec_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2024-05-08T16:21:50.183 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: sleep 1 2024-05-08T16:21:51.184 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:552: TEST_ec_backfill_unfound: for tmp in $(seq 1 240) 2024-05-08T16:21:51.185 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:553: TEST_ec_backfill_unfound: get_state 2.0 2024-05-08T16:21:51.185 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:60: get_state: local pgid=2.0 2024-05-08T16:21:51.185 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:61: get_state: local sname=state 2024-05-08T16:21:51.185 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:62: get_state: ceph --format json pg dump pgs 2024-05-08T16:21:51.185 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:63: get_state: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .state' 2024-05-08T16:21:51.601 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:553: TEST_ec_backfill_unfound: state=active+undersized+degraded+remapped+backfilling 2024-05-08T16:21:51.601 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:554: TEST_ec_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2024-05-08T16:21:51.601 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:554: TEST_ec_backfill_unfound: grep backfill_unfound 2024-05-08T16:21:51.603 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:555: TEST_ec_backfill_unfound: '[' 1 = 0 ']' 2024-05-08T16:21:51.603 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:558: TEST_ec_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2024-05-08T16:21:51.603 INFO:tasks.workunit.client.0.smithi172.stdout:active+undersized+degraded+remapped+backfilling 2024-05-08T16:21:51.603 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: sleep 1 2024-05-08T16:21:52.604 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:552: TEST_ec_backfill_unfound: for tmp in $(seq 1 240) 2024-05-08T16:21:52.605 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:553: TEST_ec_backfill_unfound: get_state 2.0 2024-05-08T16:21:52.605 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:60: get_state: local pgid=2.0 2024-05-08T16:21:52.606 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:61: get_state: local sname=state 2024-05-08T16:21:52.606 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:62: get_state: ceph --format json pg dump pgs 2024-05-08T16:21:52.606 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:63: get_state: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .state' 2024-05-08T16:21:53.034 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:553: TEST_ec_backfill_unfound: state=active+undersized+degraded+remapped+backfilling 2024-05-08T16:21:53.034 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:554: TEST_ec_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2024-05-08T16:21:53.034 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:554: TEST_ec_backfill_unfound: grep backfill_unfound 2024-05-08T16:21:53.035 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:555: TEST_ec_backfill_unfound: '[' 1 = 0 ']' 2024-05-08T16:21:53.035 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:558: TEST_ec_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2024-05-08T16:21:53.035 INFO:tasks.workunit.client.0.smithi172.stdout:active+undersized+degraded+remapped+backfilling 2024-05-08T16:21:53.036 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: sleep 1 2024-05-08T16:21:54.037 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:552: TEST_ec_backfill_unfound: for tmp in $(seq 1 240) 2024-05-08T16:21:54.037 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:553: TEST_ec_backfill_unfound: get_state 2.0 2024-05-08T16:21:54.038 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:60: get_state: local pgid=2.0 2024-05-08T16:21:54.038 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:61: get_state: local sname=state 2024-05-08T16:21:54.038 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:62: get_state: ceph --format json pg dump pgs 2024-05-08T16:21:54.038 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:63: get_state: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .state' 2024-05-08T16:21:54.463 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:553: TEST_ec_backfill_unfound: state=active+undersized+degraded+remapped+backfilling 2024-05-08T16:21:54.464 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:554: TEST_ec_backfill_unfound: grep backfill_unfound 2024-05-08T16:21:54.464 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:554: TEST_ec_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2024-05-08T16:21:54.465 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:555: TEST_ec_backfill_unfound: '[' 1 = 0 ']' 2024-05-08T16:21:54.465 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:558: TEST_ec_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2024-05-08T16:21:54.466 INFO:tasks.workunit.client.0.smithi172.stdout:active+undersized+degraded+remapped+backfilling 2024-05-08T16:21:54.466 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: sleep 1 2024-05-08T16:21:55.467 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:552: TEST_ec_backfill_unfound: for tmp in $(seq 1 240) 2024-05-08T16:21:55.468 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:553: TEST_ec_backfill_unfound: get_state 2.0 2024-05-08T16:21:55.468 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:60: get_state: local pgid=2.0 2024-05-08T16:21:55.468 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:61: get_state: local sname=state 2024-05-08T16:21:55.468 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:62: get_state: ceph --format json pg dump pgs 2024-05-08T16:21:55.468 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:63: get_state: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .state' 2024-05-08T16:21:55.883 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:553: TEST_ec_backfill_unfound: state=active+undersized+degraded+remapped+backfilling 2024-05-08T16:21:55.883 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:554: TEST_ec_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2024-05-08T16:21:55.883 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:554: TEST_ec_backfill_unfound: grep backfill_unfound 2024-05-08T16:21:55.884 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:555: TEST_ec_backfill_unfound: '[' 1 = 0 ']' 2024-05-08T16:21:55.884 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:558: TEST_ec_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2024-05-08T16:21:55.885 INFO:tasks.workunit.client.0.smithi172.stdout:active+undersized+degraded+remapped+backfilling 2024-05-08T16:21:55.885 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: sleep 1 2024-05-08T16:21:56.886 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:552: TEST_ec_backfill_unfound: for tmp in $(seq 1 240) 2024-05-08T16:21:56.887 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:553: TEST_ec_backfill_unfound: get_state 2.0 2024-05-08T16:21:56.887 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:60: get_state: local pgid=2.0 2024-05-08T16:21:56.887 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:61: get_state: local sname=state 2024-05-08T16:21:56.888 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:62: get_state: ceph --format json pg dump pgs 2024-05-08T16:21:56.888 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:63: get_state: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .state' 2024-05-08T16:21:57.288 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:553: TEST_ec_backfill_unfound: state=active+undersized+degraded+remapped+backfilling 2024-05-08T16:21:57.289 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:554: TEST_ec_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2024-05-08T16:21:57.289 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:554: TEST_ec_backfill_unfound: grep backfill_unfound 2024-05-08T16:21:57.290 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:555: TEST_ec_backfill_unfound: '[' 1 = 0 ']' 2024-05-08T16:21:57.290 INFO:tasks.workunit.client.0.smithi172.stdout:active+undersized+degraded+remapped+backfilling 2024-05-08T16:21:57.290 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:558: TEST_ec_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2024-05-08T16:21:57.291 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: sleep 1 2024-05-08T16:21:58.291 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:552: TEST_ec_backfill_unfound: for tmp in $(seq 1 240) 2024-05-08T16:21:58.292 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:553: TEST_ec_backfill_unfound: get_state 2.0 2024-05-08T16:21:58.293 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:60: get_state: local pgid=2.0 2024-05-08T16:21:58.293 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:61: get_state: local sname=state 2024-05-08T16:21:58.293 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:62: get_state: ceph --format json pg dump pgs 2024-05-08T16:21:58.293 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:63: get_state: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .state' 2024-05-08T16:21:58.703 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:553: TEST_ec_backfill_unfound: state=active+undersized+degraded+remapped+backfilling 2024-05-08T16:21:58.704 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:554: TEST_ec_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2024-05-08T16:21:58.704 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:554: TEST_ec_backfill_unfound: grep backfill_unfound 2024-05-08T16:21:58.706 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:555: TEST_ec_backfill_unfound: '[' 1 = 0 ']' 2024-05-08T16:21:58.706 INFO:tasks.workunit.client.0.smithi172.stdout:active+undersized+degraded+remapped+backfilling 2024-05-08T16:21:58.707 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:558: TEST_ec_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2024-05-08T16:21:58.707 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:559: TEST_ec_backfill_unfound: sleep 1 2024-05-08T16:21:59.707 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:552: TEST_ec_backfill_unfound: for tmp in $(seq 1 240) 2024-05-08T16:21:59.708 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:553: TEST_ec_backfill_unfound: get_state 2.0 2024-05-08T16:21:59.708 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:60: get_state: local pgid=2.0 2024-05-08T16:21:59.708 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:61: get_state: local sname=state 2024-05-08T16:21:59.709 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:62: get_state: ceph --format json pg dump pgs 2024-05-08T16:21:59.709 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:63: get_state: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .state' 2024-05-08T16:22:00.171 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:553: TEST_ec_backfill_unfound: state=active+backfill_unfound+undersized+degraded+remapped 2024-05-08T16:22:00.171 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:554: TEST_ec_backfill_unfound: echo active+backfill_unfound+undersized+degraded+remapped 2024-05-08T16:22:00.171 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:554: TEST_ec_backfill_unfound: grep backfill_unfound 2024-05-08T16:22:00.172 INFO:tasks.workunit.client.0.smithi172.stdout:active+backfill_unfound+undersized+degraded+remapped 2024-05-08T16:22:00.173 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:555: TEST_ec_backfill_unfound: '[' 0 = 0 ']' 2024-05-08T16:22:00.173 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:556: TEST_ec_backfill_unfound: break 2024-05-08T16:22:00.173 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:562: TEST_ec_backfill_unfound: ceph pg dump pgs 2024-05-08T16:22:00.540 INFO:tasks.workunit.client.0.smithi172.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-05-08T16:22:00.541 INFO:tasks.workunit.client.0.smithi172.stdout:2.0 301 1 28 0 1 1232896 0 0 101 200 101 active+backfill_unfound+undersized+degraded+remapped 2024-05-08T16:21:58.484245+0000 50'301 59:905 [3,1,4,0,2] 3 [3,1,4,0,NONE] 3 0'0 2024-05-08T16:19:38.829889+0000 0'0 2024-05-08T16:19:38.829889+0000 0 0 no scrub is scheduled 0 0 2024-05-08T16:22:00.541 INFO:tasks.workunit.client.0.smithi172.stdout:1.3 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-05-08T16:21:32.791073+0000 0'0 59:56 [4,1,2] 4 [4,1,2] 4 0'0 2024-05-08T16:16:27.197992+0000 0'0 2024-05-08T16:16:27.197992+0000 0 0 periodic scrub scheduled @ 2024-05-10T01:11:38.178191+0000 0 0 2024-05-08T16:22:00.541 INFO:tasks.workunit.client.0.smithi172.stdout:1.2 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-05-08T16:19:41.077987+0000 0'0 59:125 [0,4,1] 0 [0,4,1] 0 0'0 2024-05-08T16:16:27.197992+0000 0'0 2024-05-08T16:16:27.197992+0000 0 0 periodic scrub scheduled @ 2024-05-09T19:42:51.689645+0000 0 0 2024-05-08T16:22:00.541 INFO:tasks.workunit.client.0.smithi172.stdout:1.1 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-05-08T16:19:41.227611+0000 0'0 59:44 [4,3,0] 4 [4,3,0] 4 0'0 2024-05-08T16:16:27.197992+0000 0'0 2024-05-08T16:16:27.197992+0000 0 0 periodic scrub scheduled @ 2024-05-10T02:18:12.944334+0000 0 0 2024-05-08T16:22:00.541 INFO:tasks.workunit.client.0.smithi172.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-05-08T16:21:32.791086+0000 0'0 59:112 [1,0,2] 1 [1,0,2] 1 0'0 2024-05-08T16:16:27.197992+0000 0'0 2024-05-08T16:16:27.197992+0000 0 0 periodic scrub scheduled @ 2024-05-09T22:32:39.723989+0000 0 0 2024-05-08T16:22:00.541 INFO:tasks.workunit.client.0.smithi172.stdout: 2024-05-08T16:22:00.541 INFO:tasks.workunit.client.0.smithi172.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-05-08T16:22:00.542 INFO:tasks.workunit.client.0.smithi172.stderr:dumped pgs 2024-05-08T16:22:00.556 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:563: TEST_ec_backfill_unfound: kill_daemons td/test-erasure-eio TERM osd.2 2024-05-08T16:22:00.557 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-08T16:22:00.557 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-08T16:22:00.558 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-08T16:22:00.558 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-08T16:22:00.558 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-08T16:22:00.867 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-08T16:22:00.868 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:564: TEST_ec_backfill_unfound: sleep 5 2024-05-08T16:22:05.870 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:566: TEST_ec_backfill_unfound: ceph pg dump pgs 2024-05-08T16:22:06.263 INFO:tasks.workunit.client.0.smithi172.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-05-08T16:22:06.263 INFO:tasks.workunit.client.0.smithi172.stdout:2.0 301 1 303 0 1 1232896 0 0 101 200 101 active+recovery_unfound+undersized+degraded 2024-05-08T16:22:01.982969+0000 50'301 61:917 [3,1,4,0,NONE] 3 [3,1,4,0,NONE] 3 0'0 2024-05-08T16:19:38.829889+0000 0'0 2024-05-08T16:19:38.829889+0000 0 0 no scrub is scheduled 0 0 2024-05-08T16:22:06.263 INFO:tasks.workunit.client.0.smithi172.stdout:1.3 0 0 0 0 0 0 0 0 0 0 0 active+undersized 2024-05-08T16:22:01.656401+0000 0'0 61:63 [4,1] 4 [4,1] 4 0'0 2024-05-08T16:16:27.197992+0000 0'0 2024-05-08T16:16:27.197992+0000 0 0 no scrub is scheduled 0 0 2024-05-08T16:22:06.263 INFO:tasks.workunit.client.0.smithi172.stdout:1.2 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-05-08T16:19:41.077987+0000 0'0 61:129 [0,4,1] 0 [0,4,1] 0 0'0 2024-05-08T16:16:27.197992+0000 0'0 2024-05-08T16:16:27.197992+0000 0 0 periodic scrub scheduled @ 2024-05-09T19:42:51.689645+0000 0 0 2024-05-08T16:22:06.264 INFO:tasks.workunit.client.0.smithi172.stdout:1.1 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-05-08T16:19:41.227611+0000 0'0 61:48 [4,3,0] 4 [4,3,0] 4 0'0 2024-05-08T16:16:27.197992+0000 0'0 2024-05-08T16:16:27.197992+0000 0 0 periodic scrub scheduled @ 2024-05-10T02:18:12.944334+0000 0 0 2024-05-08T16:22:06.264 INFO:tasks.workunit.client.0.smithi172.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+undersized 2024-05-08T16:22:01.832415+0000 0'0 61:119 [1,0] 1 [1,0] 1 0'0 2024-05-08T16:16:27.197992+0000 0'0 2024-05-08T16:16:27.197992+0000 0 0 no scrub is scheduled 0 0 2024-05-08T16:22:06.264 INFO:tasks.workunit.client.0.smithi172.stdout: 2024-05-08T16:22:06.264 INFO:tasks.workunit.client.0.smithi172.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-05-08T16:22:06.264 INFO:tasks.workunit.client.0.smithi172.stderr:dumped pgs 2024-05-08T16:22:06.279 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:567: TEST_ec_backfill_unfound: ceph pg 2.0 list_unfound 2024-05-08T16:22:06.424 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2024-05-08T16:22:06.424 INFO:tasks.workunit.client.0.smithi172.stdout: "num_missing": 1, 2024-05-08T16:22:06.424 INFO:tasks.workunit.client.0.smithi172.stdout: "num_unfound": 1, 2024-05-08T16:22:06.424 INFO:tasks.workunit.client.0.smithi172.stdout: "objects": [ 2024-05-08T16:22:06.424 INFO:tasks.workunit.client.0.smithi172.stdout: { 2024-05-08T16:22:06.424 INFO:tasks.workunit.client.0.smithi172.stdout: "oid": { 2024-05-08T16:22:06.424 INFO:tasks.workunit.client.0.smithi172.stdout: "oid": "obj250", 2024-05-08T16:22:06.424 INFO:tasks.workunit.client.0.smithi172.stdout: "key": "", 2024-05-08T16:22:06.424 INFO:tasks.workunit.client.0.smithi172.stdout: "snapid": -2, 2024-05-08T16:22:06.425 INFO:tasks.workunit.client.0.smithi172.stdout: "hash": 2249616407, 2024-05-08T16:22:06.425 INFO:tasks.workunit.client.0.smithi172.stdout: "max": 0, 2024-05-08T16:22:06.425 INFO:tasks.workunit.client.0.smithi172.stdout: "pool": 2, 2024-05-08T16:22:06.425 INFO:tasks.workunit.client.0.smithi172.stdout: "namespace": "" 2024-05-08T16:22:06.425 INFO:tasks.workunit.client.0.smithi172.stdout: }, 2024-05-08T16:22:06.425 INFO:tasks.workunit.client.0.smithi172.stdout: "need": "50'251", 2024-05-08T16:22:06.425 INFO:tasks.workunit.client.0.smithi172.stdout: "have": "0'0", 2024-05-08T16:22:06.425 INFO:tasks.workunit.client.0.smithi172.stdout: "flags": "none", 2024-05-08T16:22:06.425 INFO:tasks.workunit.client.0.smithi172.stdout: "clean_regions": "clean_offsets: [], clean_omap: false, new_object: true", 2024-05-08T16:22:06.425 INFO:tasks.workunit.client.0.smithi172.stdout: "locations": [ 2024-05-08T16:22:06.425 INFO:tasks.workunit.client.0.smithi172.stdout: "0(3)", 2024-05-08T16:22:06.425 INFO:tasks.workunit.client.0.smithi172.stdout: "4(2)" 2024-05-08T16:22:06.425 INFO:tasks.workunit.client.0.smithi172.stdout: ] 2024-05-08T16:22:06.425 INFO:tasks.workunit.client.0.smithi172.stdout: } 2024-05-08T16:22:06.426 INFO:tasks.workunit.client.0.smithi172.stdout: ], 2024-05-08T16:22:06.426 INFO:tasks.workunit.client.0.smithi172.stdout: "state": "NotRecovering", 2024-05-08T16:22:06.426 INFO:tasks.workunit.client.0.smithi172.stdout: "available_might_have_unfound": true, 2024-05-08T16:22:06.426 INFO:tasks.workunit.client.0.smithi172.stdout: "might_have_unfound": [ 2024-05-08T16:22:06.426 INFO:tasks.workunit.client.0.smithi172.stdout: { 2024-05-08T16:22:06.426 INFO:tasks.workunit.client.0.smithi172.stdout: "osd": "2(4)", 2024-05-08T16:22:06.426 INFO:tasks.workunit.client.0.smithi172.stdout: "status": "osd is down" 2024-05-08T16:22:06.426 INFO:tasks.workunit.client.0.smithi172.stdout: } 2024-05-08T16:22:06.426 INFO:tasks.workunit.client.0.smithi172.stdout: ], 2024-05-08T16:22:06.426 INFO:tasks.workunit.client.0.smithi172.stdout: "more": false 2024-05-08T16:22:06.426 INFO:tasks.workunit.client.0.smithi172.stdout:} 2024-05-08T16:22:06.438 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:568: TEST_ec_backfill_unfound: ceph pg 2.0 query 2024-05-08T16:22:06.584 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2024-05-08T16:22:06.584 INFO:tasks.workunit.client.0.smithi172.stdout: "snap_trimq": "[]", 2024-05-08T16:22:06.584 INFO:tasks.workunit.client.0.smithi172.stdout: "snap_trimq_len": 0, 2024-05-08T16:22:06.584 INFO:tasks.workunit.client.0.smithi172.stdout: "state": "active+recovery_unfound+undersized+degraded", 2024-05-08T16:22:06.584 INFO:tasks.workunit.client.0.smithi172.stdout: "epoch": 61, 2024-05-08T16:22:06.584 INFO:tasks.workunit.client.0.smithi172.stdout: "up": [ 2024-05-08T16:22:06.584 INFO:tasks.workunit.client.0.smithi172.stdout: 3, 2024-05-08T16:22:06.584 INFO:tasks.workunit.client.0.smithi172.stdout: 1, 2024-05-08T16:22:06.584 INFO:tasks.workunit.client.0.smithi172.stdout: 4, 2024-05-08T16:22:06.584 INFO:tasks.workunit.client.0.smithi172.stdout: 0, 2024-05-08T16:22:06.584 INFO:tasks.workunit.client.0.smithi172.stdout: 2147483647 2024-05-08T16:22:06.584 INFO:tasks.workunit.client.0.smithi172.stdout: ], 2024-05-08T16:22:06.584 INFO:tasks.workunit.client.0.smithi172.stdout: "acting": [ 2024-05-08T16:22:06.585 INFO:tasks.workunit.client.0.smithi172.stdout: 3, 2024-05-08T16:22:06.585 INFO:tasks.workunit.client.0.smithi172.stdout: 1, 2024-05-08T16:22:06.585 INFO:tasks.workunit.client.0.smithi172.stdout: 4, 2024-05-08T16:22:06.585 INFO:tasks.workunit.client.0.smithi172.stdout: 0, 2024-05-08T16:22:06.585 INFO:tasks.workunit.client.0.smithi172.stdout: 2147483647 2024-05-08T16:22:06.585 INFO:tasks.workunit.client.0.smithi172.stdout: ], 2024-05-08T16:22:06.585 INFO:tasks.workunit.client.0.smithi172.stdout: "acting_recovery_backfill": [ 2024-05-08T16:22:06.585 INFO:tasks.workunit.client.0.smithi172.stdout: "0(3)", 2024-05-08T16:22:06.585 INFO:tasks.workunit.client.0.smithi172.stdout: "1(1)", 2024-05-08T16:22:06.585 INFO:tasks.workunit.client.0.smithi172.stdout: "3(0)", 2024-05-08T16:22:06.585 INFO:tasks.workunit.client.0.smithi172.stdout: "4(2)" 2024-05-08T16:22:06.585 INFO:tasks.workunit.client.0.smithi172.stdout: ], 2024-05-08T16:22:06.585 INFO:tasks.workunit.client.0.smithi172.stdout: "info": { 2024-05-08T16:22:06.585 INFO:tasks.workunit.client.0.smithi172.stdout: "pgid": "2.0s0", 2024-05-08T16:22:06.585 INFO:tasks.workunit.client.0.smithi172.stdout: "shared": "0", 2024-05-08T16:22:06.586 INFO:tasks.workunit.client.0.smithi172.stdout: "last_update": "50'301", 2024-05-08T16:22:06.586 INFO:tasks.workunit.client.0.smithi172.stdout: "last_complete": "50'250", 2024-05-08T16:22:06.586 INFO:tasks.workunit.client.0.smithi172.stdout: "log_tail": "50'200", 2024-05-08T16:22:06.586 INFO:tasks.workunit.client.0.smithi172.stdout: "last_user_version": 301, 2024-05-08T16:22:06.586 INFO:tasks.workunit.client.0.smithi172.stdout: "last_backfill": "MAX", 2024-05-08T16:22:06.586 INFO:tasks.workunit.client.0.smithi172.stdout: "purged_snaps": [], 2024-05-08T16:22:06.586 INFO:tasks.workunit.client.0.smithi172.stdout: "history": { 2024-05-08T16:22:06.586 INFO:tasks.workunit.client.0.smithi172.stdout: "epoch_created": 40, 2024-05-08T16:22:06.586 INFO:tasks.workunit.client.0.smithi172.stdout: "epoch_pool_created": 40, 2024-05-08T16:22:06.586 INFO:tasks.workunit.client.0.smithi172.stdout: "last_epoch_started": 61, 2024-05-08T16:22:06.586 INFO:tasks.workunit.client.0.smithi172.stdout: "last_interval_started": 60, 2024-05-08T16:22:06.586 INFO:tasks.workunit.client.0.smithi172.stdout: "last_epoch_clean": 43, 2024-05-08T16:22:06.587 INFO:tasks.workunit.client.0.smithi172.stdout: "last_interval_clean": 40, 2024-05-08T16:22:06.587 INFO:tasks.workunit.client.0.smithi172.stdout: "last_epoch_split": 0, 2024-05-08T16:22:06.587 INFO:tasks.workunit.client.0.smithi172.stdout: "last_epoch_marked_full": 0, 2024-05-08T16:22:06.587 INFO:tasks.workunit.client.0.smithi172.stdout: "same_up_since": 60, 2024-05-08T16:22:06.587 INFO:tasks.workunit.client.0.smithi172.stdout: "same_interval_since": 60, 2024-05-08T16:22:06.587 INFO:tasks.workunit.client.0.smithi172.stdout: "same_primary_since": 40, 2024-05-08T16:22:06.587 INFO:tasks.workunit.client.0.smithi172.stdout: "last_scrub": "0'0", 2024-05-08T16:22:06.587 INFO:tasks.workunit.client.0.smithi172.stdout: "last_scrub_stamp": "2024-05-08T16:19:38.829889+0000", 2024-05-08T16:22:06.587 INFO:tasks.workunit.client.0.smithi172.stdout: "last_deep_scrub": "0'0", 2024-05-08T16:22:06.587 INFO:tasks.workunit.client.0.smithi172.stdout: "last_deep_scrub_stamp": "2024-05-08T16:19:38.829889+0000", 2024-05-08T16:22:06.587 INFO:tasks.workunit.client.0.smithi172.stdout: "last_clean_scrub_stamp": "2024-05-08T16:19:38.829889+0000", 2024-05-08T16:22:06.587 INFO:tasks.workunit.client.0.smithi172.stdout: "prior_readable_until_ub": 0 2024-05-08T16:22:06.587 INFO:tasks.workunit.client.0.smithi172.stdout: }, 2024-05-08T16:22:06.587 INFO:tasks.workunit.client.0.smithi172.stdout: "stats": { 2024-05-08T16:22:06.587 INFO:tasks.workunit.client.0.smithi172.stdout: "version": "50'301", 2024-05-08T16:22:06.588 INFO:tasks.workunit.client.0.smithi172.stdout: "reported_seq": 918, 2024-05-08T16:22:06.588 INFO:tasks.workunit.client.0.smithi172.stdout: "reported_epoch": 61, 2024-05-08T16:22:06.588 INFO:tasks.workunit.client.0.smithi172.stdout: "state": "active+recovery_unfound+undersized+degraded", 2024-05-08T16:22:06.588 INFO:tasks.workunit.client.0.smithi172.stdout: "last_fresh": "2024-05-08T16:22:01.982969+0000", 2024-05-08T16:22:06.588 INFO:tasks.workunit.client.0.smithi172.stdout: "last_change": "2024-05-08T16:22:01.982969+0000", 2024-05-08T16:22:06.588 INFO:tasks.workunit.client.0.smithi172.stdout: "last_active": "2024-05-08T16:22:01.982969+0000", 2024-05-08T16:22:06.588 INFO:tasks.workunit.client.0.smithi172.stdout: "last_peered": "2024-05-08T16:22:01.982969+0000", 2024-05-08T16:22:06.588 INFO:tasks.workunit.client.0.smithi172.stdout: "last_clean": "2024-05-08T16:19:47.631736+0000", 2024-05-08T16:22:06.588 INFO:tasks.workunit.client.0.smithi172.stdout: "last_became_active": "2024-05-08T16:22:01.942110+0000", 2024-05-08T16:22:06.588 INFO:tasks.workunit.client.0.smithi172.stdout: "last_became_peered": "2024-05-08T16:22:01.942110+0000", 2024-05-08T16:22:06.588 INFO:tasks.workunit.client.0.smithi172.stdout: "last_unstale": "2024-05-08T16:22:01.982969+0000", 2024-05-08T16:22:06.588 INFO:tasks.workunit.client.0.smithi172.stdout: "last_undegraded": "2024-05-08T16:22:01.522219+0000", 2024-05-08T16:22:06.588 INFO:tasks.workunit.client.0.smithi172.stdout: "last_fullsized": "2024-05-08T16:22:01.521398+0000", 2024-05-08T16:22:06.588 INFO:tasks.workunit.client.0.smithi172.stdout: "mapping_epoch": 60, 2024-05-08T16:22:06.588 INFO:tasks.workunit.client.0.smithi172.stdout: "log_start": "50'200", 2024-05-08T16:22:06.589 INFO:tasks.workunit.client.0.smithi172.stdout: "ondisk_log_start": "50'200", 2024-05-08T16:22:06.589 INFO:tasks.workunit.client.0.smithi172.stdout: "created": 40, 2024-05-08T16:22:06.589 INFO:tasks.workunit.client.0.smithi172.stdout: "last_epoch_clean": 43, 2024-05-08T16:22:06.589 INFO:tasks.workunit.client.0.smithi172.stdout: "parent": "0.0", 2024-05-08T16:22:06.589 INFO:tasks.workunit.client.0.smithi172.stdout: "parent_split_bits": 0, 2024-05-08T16:22:06.589 INFO:tasks.workunit.client.0.smithi172.stdout: "last_scrub": "0'0", 2024-05-08T16:22:06.589 INFO:tasks.workunit.client.0.smithi172.stdout: "last_scrub_stamp": "2024-05-08T16:19:38.829889+0000", 2024-05-08T16:22:06.589 INFO:tasks.workunit.client.0.smithi172.stdout: "last_deep_scrub": "0'0", 2024-05-08T16:22:06.589 INFO:tasks.workunit.client.0.smithi172.stdout: "last_deep_scrub_stamp": "2024-05-08T16:19:38.829889+0000", 2024-05-08T16:22:06.589 INFO:tasks.workunit.client.0.smithi172.stdout: "last_clean_scrub_stamp": "2024-05-08T16:19:38.829889+0000", 2024-05-08T16:22:06.589 INFO:tasks.workunit.client.0.smithi172.stdout: "objects_scrubbed": 0, 2024-05-08T16:22:06.589 INFO:tasks.workunit.client.0.smithi172.stdout: "log_size": 101, 2024-05-08T16:22:06.589 INFO:tasks.workunit.client.0.smithi172.stdout: "log_dups_size": 200, 2024-05-08T16:22:06.589 INFO:tasks.workunit.client.0.smithi172.stdout: "ondisk_log_size": 101, 2024-05-08T16:22:06.590 INFO:tasks.workunit.client.0.smithi172.stdout: "stats_invalid": false, 2024-05-08T16:22:06.590 INFO:tasks.workunit.client.0.smithi172.stdout: "dirty_stats_invalid": false, 2024-05-08T16:22:06.590 INFO:tasks.workunit.client.0.smithi172.stdout: "omap_stats_invalid": false, 2024-05-08T16:22:06.590 INFO:tasks.workunit.client.0.smithi172.stdout: "hitset_stats_invalid": false, 2024-05-08T16:22:06.590 INFO:tasks.workunit.client.0.smithi172.stdout: "hitset_bytes_stats_invalid": false, 2024-05-08T16:22:06.590 INFO:tasks.workunit.client.0.smithi172.stdout: "pin_stats_invalid": false, 2024-05-08T16:22:06.590 INFO:tasks.workunit.client.0.smithi172.stdout: "manifest_stats_invalid": false, 2024-05-08T16:22:06.590 INFO:tasks.workunit.client.0.smithi172.stdout: "snaptrimq_len": 0, 2024-05-08T16:22:06.590 INFO:tasks.workunit.client.0.smithi172.stdout: "last_scrub_duration": 0, 2024-05-08T16:22:06.590 INFO:tasks.workunit.client.0.smithi172.stdout: "scrub_schedule": "no scrub is scheduled", 2024-05-08T16:22:06.590 INFO:tasks.workunit.client.0.smithi172.stdout: "scrub_duration": 0, 2024-05-08T16:22:06.590 INFO:tasks.workunit.client.0.smithi172.stdout: "objects_trimmed": 0, 2024-05-08T16:22:06.590 INFO:tasks.workunit.client.0.smithi172.stdout: "snaptrim_duration": 0, 2024-05-08T16:22:06.590 INFO:tasks.workunit.client.0.smithi172.stdout: "stat_sum": { 2024-05-08T16:22:06.590 INFO:tasks.workunit.client.0.smithi172.stdout: "num_bytes": 1232896, 2024-05-08T16:22:06.590 INFO:tasks.workunit.client.0.smithi172.stdout: "num_objects": 301, 2024-05-08T16:22:06.591 INFO:tasks.workunit.client.0.smithi172.stdout: "num_object_clones": 0, 2024-05-08T16:22:06.591 INFO:tasks.workunit.client.0.smithi172.stdout: "num_object_copies": 1505, 2024-05-08T16:22:06.591 INFO:tasks.workunit.client.0.smithi172.stdout: "num_objects_missing_on_primary": 1, 2024-05-08T16:22:06.591 INFO:tasks.workunit.client.0.smithi172.stdout: "num_objects_missing": 1, 2024-05-08T16:22:06.591 INFO:tasks.workunit.client.0.smithi172.stdout: "num_objects_degraded": 303, 2024-05-08T16:22:06.591 INFO:tasks.workunit.client.0.smithi172.stdout: "num_objects_misplaced": 0, 2024-05-08T16:22:06.591 INFO:tasks.workunit.client.0.smithi172.stdout: "num_objects_unfound": 1, 2024-05-08T16:22:06.591 INFO:tasks.workunit.client.0.smithi172.stdout: "num_objects_dirty": 301, 2024-05-08T16:22:06.591 INFO:tasks.workunit.client.0.smithi172.stdout: "num_whiteouts": 0, 2024-05-08T16:22:06.591 INFO:tasks.workunit.client.0.smithi172.stdout: "num_read": 0, 2024-05-08T16:22:06.591 INFO:tasks.workunit.client.0.smithi172.stdout: "num_read_kb": 0, 2024-05-08T16:22:06.591 INFO:tasks.workunit.client.0.smithi172.stdout: "num_write": 301, 2024-05-08T16:22:06.591 INFO:tasks.workunit.client.0.smithi172.stdout: "num_write_kb": 1204, 2024-05-08T16:22:06.591 INFO:tasks.workunit.client.0.smithi172.stdout: "num_scrub_errors": 0, 2024-05-08T16:22:06.592 INFO:tasks.workunit.client.0.smithi172.stdout: "num_shallow_scrub_errors": 0, 2024-05-08T16:22:06.592 INFO:tasks.workunit.client.0.smithi172.stdout: "num_deep_scrub_errors": 0, 2024-05-08T16:22:06.592 INFO:tasks.workunit.client.0.smithi172.stdout: "num_objects_recovered": 0, 2024-05-08T16:22:06.592 INFO:tasks.workunit.client.0.smithi172.stdout: "num_bytes_recovered": 0, 2024-05-08T16:22:06.592 INFO:tasks.workunit.client.0.smithi172.stdout: "num_keys_recovered": 0, 2024-05-08T16:22:06.592 INFO:tasks.workunit.client.0.smithi172.stdout: "num_objects_omap": 0, 2024-05-08T16:22:06.592 INFO:tasks.workunit.client.0.smithi172.stdout: "num_objects_hit_set_archive": 0, 2024-05-08T16:22:06.592 INFO:tasks.workunit.client.0.smithi172.stdout: "num_bytes_hit_set_archive": 0, 2024-05-08T16:22:06.592 INFO:tasks.workunit.client.0.smithi172.stdout: "num_flush": 0, 2024-05-08T16:22:06.592 INFO:tasks.workunit.client.0.smithi172.stdout: "num_flush_kb": 0, 2024-05-08T16:22:06.592 INFO:tasks.workunit.client.0.smithi172.stdout: "num_evict": 0, 2024-05-08T16:22:06.592 INFO:tasks.workunit.client.0.smithi172.stdout: "num_evict_kb": 0, 2024-05-08T16:22:06.592 INFO:tasks.workunit.client.0.smithi172.stdout: "num_promote": 0, 2024-05-08T16:22:06.592 INFO:tasks.workunit.client.0.smithi172.stdout: "num_flush_mode_high": 0, 2024-05-08T16:22:06.592 INFO:tasks.workunit.client.0.smithi172.stdout: "num_flush_mode_low": 0, 2024-05-08T16:22:06.592 INFO:tasks.workunit.client.0.smithi172.stdout: "num_evict_mode_some": 0, 2024-05-08T16:22:06.593 INFO:tasks.workunit.client.0.smithi172.stdout: "num_evict_mode_full": 0, 2024-05-08T16:22:06.593 INFO:tasks.workunit.client.0.smithi172.stdout: "num_objects_pinned": 0, 2024-05-08T16:22:06.593 INFO:tasks.workunit.client.0.smithi172.stdout: "num_legacy_snapsets": 0, 2024-05-08T16:22:06.593 INFO:tasks.workunit.client.0.smithi172.stdout: "num_large_omap_objects": 0, 2024-05-08T16:22:06.593 INFO:tasks.workunit.client.0.smithi172.stdout: "num_objects_manifest": 0, 2024-05-08T16:22:06.593 INFO:tasks.workunit.client.0.smithi172.stdout: "num_omap_bytes": 0, 2024-05-08T16:22:06.593 INFO:tasks.workunit.client.0.smithi172.stdout: "num_omap_keys": 0, 2024-05-08T16:22:06.593 INFO:tasks.workunit.client.0.smithi172.stdout: "num_objects_repaired": 0 2024-05-08T16:22:06.593 INFO:tasks.workunit.client.0.smithi172.stdout: }, 2024-05-08T16:22:06.593 INFO:tasks.workunit.client.0.smithi172.stdout: "up": [ 2024-05-08T16:22:06.593 INFO:tasks.workunit.client.0.smithi172.stdout: 3, 2024-05-08T16:22:06.593 INFO:tasks.workunit.client.0.smithi172.stdout: 1, 2024-05-08T16:22:06.593 INFO:tasks.workunit.client.0.smithi172.stdout: 4, 2024-05-08T16:22:06.593 INFO:tasks.workunit.client.0.smithi172.stdout: 0, 2024-05-08T16:22:06.593 INFO:tasks.workunit.client.0.smithi172.stdout: 2147483647 2024-05-08T16:22:06.593 INFO:tasks.workunit.client.0.smithi172.stdout: ], 2024-05-08T16:22:06.594 INFO:tasks.workunit.client.0.smithi172.stdout: "acting": [ 2024-05-08T16:22:06.594 INFO:tasks.workunit.client.0.smithi172.stdout: 3, 2024-05-08T16:22:06.594 INFO:tasks.workunit.client.0.smithi172.stdout: 1, 2024-05-08T16:22:06.594 INFO:tasks.workunit.client.0.smithi172.stdout: 4, 2024-05-08T16:22:06.594 INFO:tasks.workunit.client.0.smithi172.stdout: 0, 2024-05-08T16:22:06.594 INFO:tasks.workunit.client.0.smithi172.stdout: 2147483647 2024-05-08T16:22:06.594 INFO:tasks.workunit.client.0.smithi172.stdout: ], 2024-05-08T16:22:06.594 INFO:tasks.workunit.client.0.smithi172.stdout: "avail_no_missing": [ 2024-05-08T16:22:06.594 INFO:tasks.workunit.client.0.smithi172.stdout: "0(3)", 2024-05-08T16:22:06.594 INFO:tasks.workunit.client.0.smithi172.stdout: "4(2)" 2024-05-08T16:22:06.594 INFO:tasks.workunit.client.0.smithi172.stdout: ], 2024-05-08T16:22:06.594 INFO:tasks.workunit.client.0.smithi172.stdout: "object_location_counts": [ 2024-05-08T16:22:06.594 INFO:tasks.workunit.client.0.smithi172.stdout: { 2024-05-08T16:22:06.594 INFO:tasks.workunit.client.0.smithi172.stdout: "shards": "0(3),1(1),3(0),4(2)", 2024-05-08T16:22:06.595 INFO:tasks.workunit.client.0.smithi172.stdout: "objects": 300 2024-05-08T16:22:06.595 INFO:tasks.workunit.client.0.smithi172.stdout: }, 2024-05-08T16:22:06.595 INFO:tasks.workunit.client.0.smithi172.stdout: { 2024-05-08T16:22:06.595 INFO:tasks.workunit.client.0.smithi172.stdout: "shards": "0(3),4(2)", 2024-05-08T16:22:06.595 INFO:tasks.workunit.client.0.smithi172.stdout: "objects": 1 2024-05-08T16:22:06.595 INFO:tasks.workunit.client.0.smithi172.stdout: } 2024-05-08T16:22:06.595 INFO:tasks.workunit.client.0.smithi172.stdout: ], 2024-05-08T16:22:06.595 INFO:tasks.workunit.client.0.smithi172.stdout: "blocked_by": [], 2024-05-08T16:22:06.595 INFO:tasks.workunit.client.0.smithi172.stdout: "up_primary": 3, 2024-05-08T16:22:06.595 INFO:tasks.workunit.client.0.smithi172.stdout: "acting_primary": 3, 2024-05-08T16:22:06.595 INFO:tasks.workunit.client.0.smithi172.stdout: "purged_snaps": [] 2024-05-08T16:22:06.595 INFO:tasks.workunit.client.0.smithi172.stdout: }, 2024-05-08T16:22:06.595 INFO:tasks.workunit.client.0.smithi172.stdout: "empty": 0, 2024-05-08T16:22:06.595 INFO:tasks.workunit.client.0.smithi172.stdout: "dne": 0, 2024-05-08T16:22:06.595 INFO:tasks.workunit.client.0.smithi172.stdout: "incomplete": 0, 2024-05-08T16:22:06.595 INFO:tasks.workunit.client.0.smithi172.stdout: "last_epoch_started": 61, 2024-05-08T16:22:06.596 INFO:tasks.workunit.client.0.smithi172.stdout: "hit_set_history": { 2024-05-08T16:22:06.596 INFO:tasks.workunit.client.0.smithi172.stdout: "current_last_update": "0'0", 2024-05-08T16:22:06.596 INFO:tasks.workunit.client.0.smithi172.stdout: "history": [] 2024-05-08T16:22:06.596 INFO:tasks.workunit.client.0.smithi172.stdout: } 2024-05-08T16:22:06.596 INFO:tasks.workunit.client.0.smithi172.stdout: }, 2024-05-08T16:22:06.596 INFO:tasks.workunit.client.0.smithi172.stdout: "peer_info": [ 2024-05-08T16:22:06.596 INFO:tasks.workunit.client.0.smithi172.stdout: { 2024-05-08T16:22:06.596 INFO:tasks.workunit.client.0.smithi172.stdout: "peer": "0(3)", 2024-05-08T16:22:06.596 INFO:tasks.workunit.client.0.smithi172.stdout: "pgid": "2.0s3", 2024-05-08T16:22:06.596 INFO:tasks.workunit.client.0.smithi172.stdout: "shared": "3", 2024-05-08T16:22:06.596 INFO:tasks.workunit.client.0.smithi172.stdout: "last_update": "50'301", 2024-05-08T16:22:06.596 INFO:tasks.workunit.client.0.smithi172.stdout: "last_complete": "50'301", 2024-05-08T16:22:06.596 INFO:tasks.workunit.client.0.smithi172.stdout: "log_tail": "50'200", 2024-05-08T16:22:06.596 INFO:tasks.workunit.client.0.smithi172.stdout: "last_user_version": 301, 2024-05-08T16:22:06.597 INFO:tasks.workunit.client.0.smithi172.stdout: "last_backfill": "MAX", 2024-05-08T16:22:06.597 INFO:tasks.workunit.client.0.smithi172.stdout: "purged_snaps": [], 2024-05-08T16:22:06.597 INFO:tasks.workunit.client.0.smithi172.stdout: "history": { 2024-05-08T16:22:06.597 INFO:tasks.workunit.client.0.smithi172.stdout: "epoch_created": 40, 2024-05-08T16:22:06.597 INFO:tasks.workunit.client.0.smithi172.stdout: "epoch_pool_created": 40, 2024-05-08T16:22:06.597 INFO:tasks.workunit.client.0.smithi172.stdout: "last_epoch_started": 61, 2024-05-08T16:22:06.597 INFO:tasks.workunit.client.0.smithi172.stdout: "last_interval_started": 60, 2024-05-08T16:22:06.597 INFO:tasks.workunit.client.0.smithi172.stdout: "last_epoch_clean": 43, 2024-05-08T16:22:06.597 INFO:tasks.workunit.client.0.smithi172.stdout: "last_interval_clean": 40, 2024-05-08T16:22:06.597 INFO:tasks.workunit.client.0.smithi172.stdout: "last_epoch_split": 0, 2024-05-08T16:22:06.597 INFO:tasks.workunit.client.0.smithi172.stdout: "last_epoch_marked_full": 0, 2024-05-08T16:22:06.597 INFO:tasks.workunit.client.0.smithi172.stdout: "same_up_since": 60, 2024-05-08T16:22:06.597 INFO:tasks.workunit.client.0.smithi172.stdout: "same_interval_since": 60, 2024-05-08T16:22:06.597 INFO:tasks.workunit.client.0.smithi172.stdout: "same_primary_since": 40, 2024-05-08T16:22:06.597 INFO:tasks.workunit.client.0.smithi172.stdout: "last_scrub": "0'0", 2024-05-08T16:22:06.598 INFO:tasks.workunit.client.0.smithi172.stdout: "last_scrub_stamp": "2024-05-08T16:19:38.829889+0000", 2024-05-08T16:22:06.598 INFO:tasks.workunit.client.0.smithi172.stdout: "last_deep_scrub": "0'0", 2024-05-08T16:22:06.598 INFO:tasks.workunit.client.0.smithi172.stdout: "last_deep_scrub_stamp": "2024-05-08T16:19:38.829889+0000", 2024-05-08T16:22:06.598 INFO:tasks.workunit.client.0.smithi172.stdout: "last_clean_scrub_stamp": "2024-05-08T16:19:38.829889+0000", 2024-05-08T16:22:06.598 INFO:tasks.workunit.client.0.smithi172.stdout: "prior_readable_until_ub": 0 2024-05-08T16:22:06.598 INFO:tasks.workunit.client.0.smithi172.stdout: }, 2024-05-08T16:22:06.598 INFO:tasks.workunit.client.0.smithi172.stdout: "stats": { 2024-05-08T16:22:06.598 INFO:tasks.workunit.client.0.smithi172.stdout: "version": "50'301", 2024-05-08T16:22:06.598 INFO:tasks.workunit.client.0.smithi172.stdout: "reported_seq": 908, 2024-05-08T16:22:06.598 INFO:tasks.workunit.client.0.smithi172.stdout: "reported_epoch": 60, 2024-05-08T16:22:06.598 INFO:tasks.workunit.client.0.smithi172.stdout: "state": "active+undersized+degraded", 2024-05-08T16:22:06.598 INFO:tasks.workunit.client.0.smithi172.stdout: "last_fresh": "2024-05-08T16:21:23.637408+0000", 2024-05-08T16:22:06.598 INFO:tasks.workunit.client.0.smithi172.stdout: "last_change": "2024-05-08T16:19:50.127512+0000", 2024-05-08T16:22:06.598 INFO:tasks.workunit.client.0.smithi172.stdout: "last_active": "2024-05-08T16:21:23.637408+0000", 2024-05-08T16:22:06.598 INFO:tasks.workunit.client.0.smithi172.stdout: "last_peered": "2024-05-08T16:21:23.637408+0000", 2024-05-08T16:22:06.598 INFO:tasks.workunit.client.0.smithi172.stdout: "last_clean": "2024-05-08T16:19:47.631736+0000", 2024-05-08T16:22:06.599 INFO:tasks.workunit.client.0.smithi172.stdout: "last_became_active": "2024-05-08T16:19:50.127512+0000", 2024-05-08T16:22:06.599 INFO:tasks.workunit.client.0.smithi172.stdout: "last_became_peered": "2024-05-08T16:19:50.127512+0000", 2024-05-08T16:22:06.599 INFO:tasks.workunit.client.0.smithi172.stdout: "last_unstale": "2024-05-08T16:21:23.637408+0000", 2024-05-08T16:22:06.599 INFO:tasks.workunit.client.0.smithi172.stdout: "last_undegraded": "2024-05-08T16:19:49.766134+0000", 2024-05-08T16:22:06.599 INFO:tasks.workunit.client.0.smithi172.stdout: "last_fullsized": "2024-05-08T16:19:49.765821+0000", 2024-05-08T16:22:06.599 INFO:tasks.workunit.client.0.smithi172.stdout: "mapping_epoch": 60, 2024-05-08T16:22:06.599 INFO:tasks.workunit.client.0.smithi172.stdout: "log_start": "50'200", 2024-05-08T16:22:06.599 INFO:tasks.workunit.client.0.smithi172.stdout: "ondisk_log_start": "50'200", 2024-05-08T16:22:06.599 INFO:tasks.workunit.client.0.smithi172.stdout: "created": 40, 2024-05-08T16:22:06.599 INFO:tasks.workunit.client.0.smithi172.stdout: "last_epoch_clean": 43, 2024-05-08T16:22:06.599 INFO:tasks.workunit.client.0.smithi172.stdout: "parent": "0.0", 2024-05-08T16:22:06.599 INFO:tasks.workunit.client.0.smithi172.stdout: "parent_split_bits": 0, 2024-05-08T16:22:06.599 INFO:tasks.workunit.client.0.smithi172.stdout: "last_scrub": "0'0", 2024-05-08T16:22:06.599 INFO:tasks.workunit.client.0.smithi172.stdout: "last_scrub_stamp": "2024-05-08T16:19:38.829889+0000", 2024-05-08T16:22:06.599 INFO:tasks.workunit.client.0.smithi172.stdout: "last_deep_scrub": "0'0", 2024-05-08T16:22:06.600 INFO:tasks.workunit.client.0.smithi172.stdout: "last_deep_scrub_stamp": "2024-05-08T16:19:38.829889+0000", 2024-05-08T16:22:06.600 INFO:tasks.workunit.client.0.smithi172.stdout: "last_clean_scrub_stamp": "2024-05-08T16:19:38.829889+0000", 2024-05-08T16:22:06.600 INFO:tasks.workunit.client.0.smithi172.stdout: "objects_scrubbed": 0, 2024-05-08T16:22:06.600 INFO:tasks.workunit.client.0.smithi172.stdout: "log_size": 101, 2024-05-08T16:22:06.600 INFO:tasks.workunit.client.0.smithi172.stdout: "log_dups_size": 200, 2024-05-08T16:22:06.600 INFO:tasks.workunit.client.0.smithi172.stdout: "ondisk_log_size": 101, 2024-05-08T16:22:06.600 INFO:tasks.workunit.client.0.smithi172.stdout: "stats_invalid": false, 2024-05-08T16:22:06.600 INFO:tasks.workunit.client.0.smithi172.stdout: "dirty_stats_invalid": false, 2024-05-08T16:22:06.600 INFO:tasks.workunit.client.0.smithi172.stdout: "omap_stats_invalid": false, 2024-05-08T16:22:06.600 INFO:tasks.workunit.client.0.smithi172.stdout: "hitset_stats_invalid": false, 2024-05-08T16:22:06.600 INFO:tasks.workunit.client.0.smithi172.stdout: "hitset_bytes_stats_invalid": false, 2024-05-08T16:22:06.600 INFO:tasks.workunit.client.0.smithi172.stdout: "pin_stats_invalid": false, 2024-05-08T16:22:06.600 INFO:tasks.workunit.client.0.smithi172.stdout: "manifest_stats_invalid": false, 2024-05-08T16:22:06.600 INFO:tasks.workunit.client.0.smithi172.stdout: "snaptrimq_len": 0, 2024-05-08T16:22:06.600 INFO:tasks.workunit.client.0.smithi172.stdout: "last_scrub_duration": 0, 2024-05-08T16:22:06.601 INFO:tasks.workunit.client.0.smithi172.stdout: "scrub_schedule": "periodic scrub scheduled @ 2024-05-09T17:43:21.349481+0000", 2024-05-08T16:22:06.601 INFO:tasks.workunit.client.0.smithi172.stdout: "scrub_duration": 0, 2024-05-08T16:22:06.601 INFO:tasks.workunit.client.0.smithi172.stdout: "objects_trimmed": 0, 2024-05-08T16:22:06.601 INFO:tasks.workunit.client.0.smithi172.stdout: "snaptrim_duration": 0, 2024-05-08T16:22:06.601 INFO:tasks.workunit.client.0.smithi172.stdout: "stat_sum": { 2024-05-08T16:22:06.601 INFO:tasks.workunit.client.0.smithi172.stdout: "num_bytes": 1232896, 2024-05-08T16:22:06.601 INFO:tasks.workunit.client.0.smithi172.stdout: "num_objects": 301, 2024-05-08T16:22:06.601 INFO:tasks.workunit.client.0.smithi172.stdout: "num_object_clones": 0, 2024-05-08T16:22:06.601 INFO:tasks.workunit.client.0.smithi172.stdout: "num_object_copies": 1505, 2024-05-08T16:22:06.601 INFO:tasks.workunit.client.0.smithi172.stdout: "num_objects_missing_on_primary": 0, 2024-05-08T16:22:06.601 INFO:tasks.workunit.client.0.smithi172.stdout: "num_objects_missing": 0, 2024-05-08T16:22:06.601 INFO:tasks.workunit.client.0.smithi172.stdout: "num_objects_degraded": 301, 2024-05-08T16:22:06.601 INFO:tasks.workunit.client.0.smithi172.stdout: "num_objects_misplaced": 0, 2024-05-08T16:22:06.601 INFO:tasks.workunit.client.0.smithi172.stdout: "num_objects_unfound": 0, 2024-05-08T16:22:06.601 INFO:tasks.workunit.client.0.smithi172.stdout: "num_objects_dirty": 301, 2024-05-08T16:22:06.601 INFO:tasks.workunit.client.0.smithi172.stdout: "num_whiteouts": 0, 2024-05-08T16:22:06.602 INFO:tasks.workunit.client.0.smithi172.stdout: "num_read": 0, 2024-05-08T16:22:06.602 INFO:tasks.workunit.client.0.smithi172.stdout: "num_read_kb": 0, 2024-05-08T16:22:06.602 INFO:tasks.workunit.client.0.smithi172.stdout: "num_write": 301, 2024-05-08T16:22:06.602 INFO:tasks.workunit.client.0.smithi172.stdout: "num_write_kb": 1204, 2024-05-08T16:22:06.602 INFO:tasks.workunit.client.0.smithi172.stdout: "num_scrub_errors": 0, 2024-05-08T16:22:06.602 INFO:tasks.workunit.client.0.smithi172.stdout: "num_shallow_scrub_errors": 0, 2024-05-08T16:22:06.602 INFO:tasks.workunit.client.0.smithi172.stdout: "num_deep_scrub_errors": 0, 2024-05-08T16:22:06.602 INFO:tasks.workunit.client.0.smithi172.stdout: "num_objects_recovered": 0, 2024-05-08T16:22:06.602 INFO:tasks.workunit.client.0.smithi172.stdout: "num_bytes_recovered": 0, 2024-05-08T16:22:06.602 INFO:tasks.workunit.client.0.smithi172.stdout: "num_keys_recovered": 0, 2024-05-08T16:22:06.602 INFO:tasks.workunit.client.0.smithi172.stdout: "num_objects_omap": 0, 2024-05-08T16:22:06.602 INFO:tasks.workunit.client.0.smithi172.stdout: "num_objects_hit_set_archive": 0, 2024-05-08T16:22:06.602 INFO:tasks.workunit.client.0.smithi172.stdout: "num_bytes_hit_set_archive": 0, 2024-05-08T16:22:06.602 INFO:tasks.workunit.client.0.smithi172.stdout: "num_flush": 0, 2024-05-08T16:22:06.602 INFO:tasks.workunit.client.0.smithi172.stdout: "num_flush_kb": 0, 2024-05-08T16:22:06.603 INFO:tasks.workunit.client.0.smithi172.stdout: "num_evict": 0, 2024-05-08T16:22:06.603 INFO:tasks.workunit.client.0.smithi172.stdout: "num_evict_kb": 0, 2024-05-08T16:22:06.603 INFO:tasks.workunit.client.0.smithi172.stdout: "num_promote": 0, 2024-05-08T16:22:06.603 INFO:tasks.workunit.client.0.smithi172.stdout: "num_flush_mode_high": 0, 2024-05-08T16:22:06.603 INFO:tasks.workunit.client.0.smithi172.stdout: "num_flush_mode_low": 0, 2024-05-08T16:22:06.603 INFO:tasks.workunit.client.0.smithi172.stdout: "num_evict_mode_some": 0, 2024-05-08T16:22:06.603 INFO:tasks.workunit.client.0.smithi172.stdout: "num_evict_mode_full": 0, 2024-05-08T16:22:06.603 INFO:tasks.workunit.client.0.smithi172.stdout: "num_objects_pinned": 0, 2024-05-08T16:22:06.603 INFO:tasks.workunit.client.0.smithi172.stdout: "num_legacy_snapsets": 0, 2024-05-08T16:22:06.603 INFO:tasks.workunit.client.0.smithi172.stdout: "num_large_omap_objects": 0, 2024-05-08T16:22:06.603 INFO:tasks.workunit.client.0.smithi172.stdout: "num_objects_manifest": 0, 2024-05-08T16:22:06.603 INFO:tasks.workunit.client.0.smithi172.stdout: "num_omap_bytes": 0, 2024-05-08T16:22:06.603 INFO:tasks.workunit.client.0.smithi172.stdout: "num_omap_keys": 0, 2024-05-08T16:22:06.603 INFO:tasks.workunit.client.0.smithi172.stdout: "num_objects_repaired": 0 2024-05-08T16:22:06.603 INFO:tasks.workunit.client.0.smithi172.stdout: }, 2024-05-08T16:22:06.603 INFO:tasks.workunit.client.0.smithi172.stdout: "up": [ 2024-05-08T16:22:06.604 INFO:tasks.workunit.client.0.smithi172.stdout: 3, 2024-05-08T16:22:06.604 INFO:tasks.workunit.client.0.smithi172.stdout: 1, 2024-05-08T16:22:06.604 INFO:tasks.workunit.client.0.smithi172.stdout: 4, 2024-05-08T16:22:06.604 INFO:tasks.workunit.client.0.smithi172.stdout: 0, 2024-05-08T16:22:06.604 INFO:tasks.workunit.client.0.smithi172.stdout: 2147483647 2024-05-08T16:22:06.604 INFO:tasks.workunit.client.0.smithi172.stdout: ], 2024-05-08T16:22:06.604 INFO:tasks.workunit.client.0.smithi172.stdout: "acting": [ 2024-05-08T16:22:06.604 INFO:tasks.workunit.client.0.smithi172.stdout: 3, 2024-05-08T16:22:06.604 INFO:tasks.workunit.client.0.smithi172.stdout: 1, 2024-05-08T16:22:06.604 INFO:tasks.workunit.client.0.smithi172.stdout: 4, 2024-05-08T16:22:06.604 INFO:tasks.workunit.client.0.smithi172.stdout: 0, 2024-05-08T16:22:06.604 INFO:tasks.workunit.client.0.smithi172.stdout: 2147483647 2024-05-08T16:22:06.604 INFO:tasks.workunit.client.0.smithi172.stdout: ], 2024-05-08T16:22:06.604 INFO:tasks.workunit.client.0.smithi172.stdout: "avail_no_missing": [ 2024-05-08T16:22:06.604 INFO:tasks.workunit.client.0.smithi172.stdout: "3(0)", 2024-05-08T16:22:06.604 INFO:tasks.workunit.client.0.smithi172.stdout: "0(3)", 2024-05-08T16:22:06.605 INFO:tasks.workunit.client.0.smithi172.stdout: "1(1)", 2024-05-08T16:22:06.605 INFO:tasks.workunit.client.0.smithi172.stdout: "4(2)" 2024-05-08T16:22:06.605 INFO:tasks.workunit.client.0.smithi172.stdout: ], 2024-05-08T16:22:06.605 INFO:tasks.workunit.client.0.smithi172.stdout: "object_location_counts": [ 2024-05-08T16:22:06.605 INFO:tasks.workunit.client.0.smithi172.stdout: { 2024-05-08T16:22:06.605 INFO:tasks.workunit.client.0.smithi172.stdout: "shards": "0(3),1(1),3(0),4(2)", 2024-05-08T16:22:06.605 INFO:tasks.workunit.client.0.smithi172.stdout: "objects": 301 2024-05-08T16:22:06.605 INFO:tasks.workunit.client.0.smithi172.stdout: } 2024-05-08T16:22:06.605 INFO:tasks.workunit.client.0.smithi172.stdout: ], 2024-05-08T16:22:06.605 INFO:tasks.workunit.client.0.smithi172.stdout: "blocked_by": [], 2024-05-08T16:22:06.605 INFO:tasks.workunit.client.0.smithi172.stdout: "up_primary": 3, 2024-05-08T16:22:06.605 INFO:tasks.workunit.client.0.smithi172.stdout: "acting_primary": 3, 2024-05-08T16:22:06.605 INFO:tasks.workunit.client.0.smithi172.stdout: "purged_snaps": [] 2024-05-08T16:22:06.605 INFO:tasks.workunit.client.0.smithi172.stdout: }, 2024-05-08T16:22:06.605 INFO:tasks.workunit.client.0.smithi172.stdout: "empty": 0, 2024-05-08T16:22:06.605 INFO:tasks.workunit.client.0.smithi172.stdout: "dne": 0, 2024-05-08T16:22:06.606 INFO:tasks.workunit.client.0.smithi172.stdout: "incomplete": 0, 2024-05-08T16:22:06.606 INFO:tasks.workunit.client.0.smithi172.stdout: "last_epoch_started": 61, 2024-05-08T16:22:06.606 INFO:tasks.workunit.client.0.smithi172.stdout: "hit_set_history": { 2024-05-08T16:22:06.606 INFO:tasks.workunit.client.0.smithi172.stdout: "current_last_update": "0'0", 2024-05-08T16:22:06.606 INFO:tasks.workunit.client.0.smithi172.stdout: "history": [] 2024-05-08T16:22:06.606 INFO:tasks.workunit.client.0.smithi172.stdout: } 2024-05-08T16:22:06.606 INFO:tasks.workunit.client.0.smithi172.stdout: }, 2024-05-08T16:22:06.606 INFO:tasks.workunit.client.0.smithi172.stdout: { 2024-05-08T16:22:06.606 INFO:tasks.workunit.client.0.smithi172.stdout: "peer": "1(1)", 2024-05-08T16:22:06.606 INFO:tasks.workunit.client.0.smithi172.stdout: "pgid": "2.0s1", 2024-05-08T16:22:06.606 INFO:tasks.workunit.client.0.smithi172.stdout: "shared": "1", 2024-05-08T16:22:06.606 INFO:tasks.workunit.client.0.smithi172.stdout: "last_update": "50'301", 2024-05-08T16:22:06.606 INFO:tasks.workunit.client.0.smithi172.stdout: "last_complete": "50'301", 2024-05-08T16:22:06.606 INFO:tasks.workunit.client.0.smithi172.stdout: "log_tail": "50'200", 2024-05-08T16:22:06.606 INFO:tasks.workunit.client.0.smithi172.stdout: "last_user_version": 301, 2024-05-08T16:22:06.607 INFO:tasks.workunit.client.0.smithi172.stdout: "last_backfill": "MAX", 2024-05-08T16:22:06.607 INFO:tasks.workunit.client.0.smithi172.stdout: "purged_snaps": [], 2024-05-08T16:22:06.607 INFO:tasks.workunit.client.0.smithi172.stdout: "history": { 2024-05-08T16:22:06.607 INFO:tasks.workunit.client.0.smithi172.stdout: "epoch_created": 40, 2024-05-08T16:22:06.607 INFO:tasks.workunit.client.0.smithi172.stdout: "epoch_pool_created": 40, 2024-05-08T16:22:06.607 INFO:tasks.workunit.client.0.smithi172.stdout: "last_epoch_started": 61, 2024-05-08T16:22:06.607 INFO:tasks.workunit.client.0.smithi172.stdout: "last_interval_started": 60, 2024-05-08T16:22:06.607 INFO:tasks.workunit.client.0.smithi172.stdout: "last_epoch_clean": 43, 2024-05-08T16:22:06.607 INFO:tasks.workunit.client.0.smithi172.stdout: "last_interval_clean": 40, 2024-05-08T16:22:06.607 INFO:tasks.workunit.client.0.smithi172.stdout: "last_epoch_split": 0, 2024-05-08T16:22:06.607 INFO:tasks.workunit.client.0.smithi172.stdout: "last_epoch_marked_full": 0, 2024-05-08T16:22:06.607 INFO:tasks.workunit.client.0.smithi172.stdout: "same_up_since": 60, 2024-05-08T16:22:06.607 INFO:tasks.workunit.client.0.smithi172.stdout: "same_interval_since": 60, 2024-05-08T16:22:06.607 INFO:tasks.workunit.client.0.smithi172.stdout: "same_primary_since": 40, 2024-05-08T16:22:06.607 INFO:tasks.workunit.client.0.smithi172.stdout: "last_scrub": "0'0", 2024-05-08T16:22:06.608 INFO:tasks.workunit.client.0.smithi172.stdout: "last_scrub_stamp": "2024-05-08T16:19:38.829889+0000", 2024-05-08T16:22:06.608 INFO:tasks.workunit.client.0.smithi172.stdout: "last_deep_scrub": "0'0", 2024-05-08T16:22:06.608 INFO:tasks.workunit.client.0.smithi172.stdout: "last_deep_scrub_stamp": "2024-05-08T16:19:38.829889+0000", 2024-05-08T16:22:06.608 INFO:tasks.workunit.client.0.smithi172.stdout: "last_clean_scrub_stamp": "2024-05-08T16:19:38.829889+0000", 2024-05-08T16:22:06.608 INFO:tasks.workunit.client.0.smithi172.stdout: "prior_readable_until_ub": 0 2024-05-08T16:22:06.608 INFO:tasks.workunit.client.0.smithi172.stdout: }, 2024-05-08T16:22:06.608 INFO:tasks.workunit.client.0.smithi172.stdout: "stats": { 2024-05-08T16:22:06.608 INFO:tasks.workunit.client.0.smithi172.stdout: "version": "50'301", 2024-05-08T16:22:06.608 INFO:tasks.workunit.client.0.smithi172.stdout: "reported_seq": 336, 2024-05-08T16:22:06.608 INFO:tasks.workunit.client.0.smithi172.stdout: "reported_epoch": 50, 2024-05-08T16:22:06.608 INFO:tasks.workunit.client.0.smithi172.stdout: "state": "active+undersized+degraded", 2024-05-08T16:22:06.608 INFO:tasks.workunit.client.0.smithi172.stdout: "last_fresh": "2024-05-08T16:21:23.637408+0000", 2024-05-08T16:22:06.608 INFO:tasks.workunit.client.0.smithi172.stdout: "last_change": "2024-05-08T16:19:50.127512+0000", 2024-05-08T16:22:06.608 INFO:tasks.workunit.client.0.smithi172.stdout: "last_active": "2024-05-08T16:21:23.637408+0000", 2024-05-08T16:22:06.608 INFO:tasks.workunit.client.0.smithi172.stdout: "last_peered": "2024-05-08T16:21:23.637408+0000", 2024-05-08T16:22:06.609 INFO:tasks.workunit.client.0.smithi172.stdout: "last_clean": "2024-05-08T16:19:47.631736+0000", 2024-05-08T16:22:06.609 INFO:tasks.workunit.client.0.smithi172.stdout: "last_became_active": "2024-05-08T16:19:50.127512+0000", 2024-05-08T16:22:06.609 INFO:tasks.workunit.client.0.smithi172.stdout: "last_became_peered": "2024-05-08T16:19:50.127512+0000", 2024-05-08T16:22:06.609 INFO:tasks.workunit.client.0.smithi172.stdout: "last_unstale": "2024-05-08T16:21:23.637408+0000", 2024-05-08T16:22:06.609 INFO:tasks.workunit.client.0.smithi172.stdout: "last_undegraded": "2024-05-08T16:19:49.766134+0000", 2024-05-08T16:22:06.609 INFO:tasks.workunit.client.0.smithi172.stdout: "last_fullsized": "2024-05-08T16:19:49.765821+0000", 2024-05-08T16:22:06.609 INFO:tasks.workunit.client.0.smithi172.stdout: "mapping_epoch": 60, 2024-05-08T16:22:06.609 INFO:tasks.workunit.client.0.smithi172.stdout: "log_start": "50'200", 2024-05-08T16:22:06.609 INFO:tasks.workunit.client.0.smithi172.stdout: "ondisk_log_start": "50'200", 2024-05-08T16:22:06.609 INFO:tasks.workunit.client.0.smithi172.stdout: "created": 40, 2024-05-08T16:22:06.609 INFO:tasks.workunit.client.0.smithi172.stdout: "last_epoch_clean": 43, 2024-05-08T16:22:06.609 INFO:tasks.workunit.client.0.smithi172.stdout: "parent": "0.0", 2024-05-08T16:22:06.609 INFO:tasks.workunit.client.0.smithi172.stdout: "parent_split_bits": 0, 2024-05-08T16:22:06.609 INFO:tasks.workunit.client.0.smithi172.stdout: "last_scrub": "0'0", 2024-05-08T16:22:06.609 INFO:tasks.workunit.client.0.smithi172.stdout: "last_scrub_stamp": "2024-05-08T16:19:38.829889+0000", 2024-05-08T16:22:06.609 INFO:tasks.workunit.client.0.smithi172.stdout: "last_deep_scrub": "0'0", 2024-05-08T16:22:06.610 INFO:tasks.workunit.client.0.smithi172.stdout: "last_deep_scrub_stamp": "2024-05-08T16:19:38.829889+0000", 2024-05-08T16:22:06.610 INFO:tasks.workunit.client.0.smithi172.stdout: "last_clean_scrub_stamp": "2024-05-08T16:19:38.829889+0000", 2024-05-08T16:22:06.610 INFO:tasks.workunit.client.0.smithi172.stdout: "objects_scrubbed": 0, 2024-05-08T16:22:06.610 INFO:tasks.workunit.client.0.smithi172.stdout: "log_size": 101, 2024-05-08T16:22:06.610 INFO:tasks.workunit.client.0.smithi172.stdout: "log_dups_size": 200, 2024-05-08T16:22:06.610 INFO:tasks.workunit.client.0.smithi172.stdout: "ondisk_log_size": 101, 2024-05-08T16:22:06.610 INFO:tasks.workunit.client.0.smithi172.stdout: "stats_invalid": false, 2024-05-08T16:22:06.610 INFO:tasks.workunit.client.0.smithi172.stdout: "dirty_stats_invalid": false, 2024-05-08T16:22:06.610 INFO:tasks.workunit.client.0.smithi172.stdout: "omap_stats_invalid": false, 2024-05-08T16:22:06.610 INFO:tasks.workunit.client.0.smithi172.stdout: "hitset_stats_invalid": false, 2024-05-08T16:22:06.610 INFO:tasks.workunit.client.0.smithi172.stdout: "hitset_bytes_stats_invalid": false, 2024-05-08T16:22:06.610 INFO:tasks.workunit.client.0.smithi172.stdout: "pin_stats_invalid": false, 2024-05-08T16:22:06.610 INFO:tasks.workunit.client.0.smithi172.stdout: "manifest_stats_invalid": false, 2024-05-08T16:22:06.610 INFO:tasks.workunit.client.0.smithi172.stdout: "snaptrimq_len": 0, 2024-05-08T16:22:06.610 INFO:tasks.workunit.client.0.smithi172.stdout: "last_scrub_duration": 0, 2024-05-08T16:22:06.610 INFO:tasks.workunit.client.0.smithi172.stdout: "scrub_schedule": "periodic scrub scheduled @ 2024-05-09T17:43:21.349481+0000", 2024-05-08T16:22:06.611 INFO:tasks.workunit.client.0.smithi172.stdout: "scrub_duration": 0, 2024-05-08T16:22:06.611 INFO:tasks.workunit.client.0.smithi172.stdout: "objects_trimmed": 0, 2024-05-08T16:22:06.611 INFO:tasks.workunit.client.0.smithi172.stdout: "snaptrim_duration": 0, 2024-05-08T16:22:06.611 INFO:tasks.workunit.client.0.smithi172.stdout: "stat_sum": { 2024-05-08T16:22:06.611 INFO:tasks.workunit.client.0.smithi172.stdout: "num_bytes": 1232896, 2024-05-08T16:22:06.611 INFO:tasks.workunit.client.0.smithi172.stdout: "num_objects": 301, 2024-05-08T16:22:06.611 INFO:tasks.workunit.client.0.smithi172.stdout: "num_object_clones": 0, 2024-05-08T16:22:06.611 INFO:tasks.workunit.client.0.smithi172.stdout: "num_object_copies": 1505, 2024-05-08T16:22:06.611 INFO:tasks.workunit.client.0.smithi172.stdout: "num_objects_missing_on_primary": 0, 2024-05-08T16:22:06.611 INFO:tasks.workunit.client.0.smithi172.stdout: "num_objects_missing": 1, 2024-05-08T16:22:06.611 INFO:tasks.workunit.client.0.smithi172.stdout: "num_objects_degraded": 301, 2024-05-08T16:22:06.611 INFO:tasks.workunit.client.0.smithi172.stdout: "num_objects_misplaced": 0, 2024-05-08T16:22:06.611 INFO:tasks.workunit.client.0.smithi172.stdout: "num_objects_unfound": 0, 2024-05-08T16:22:06.611 INFO:tasks.workunit.client.0.smithi172.stdout: "num_objects_dirty": 301, 2024-05-08T16:22:06.611 INFO:tasks.workunit.client.0.smithi172.stdout: "num_whiteouts": 0, 2024-05-08T16:22:06.612 INFO:tasks.workunit.client.0.smithi172.stdout: "num_read": 0, 2024-05-08T16:22:06.612 INFO:tasks.workunit.client.0.smithi172.stdout: "num_read_kb": 0, 2024-05-08T16:22:06.612 INFO:tasks.workunit.client.0.smithi172.stdout: "num_write": 301, 2024-05-08T16:22:06.612 INFO:tasks.workunit.client.0.smithi172.stdout: "num_write_kb": 1204, 2024-05-08T16:22:06.612 INFO:tasks.workunit.client.0.smithi172.stdout: "num_scrub_errors": 0, 2024-05-08T16:22:06.612 INFO:tasks.workunit.client.0.smithi172.stdout: "num_shallow_scrub_errors": 0, 2024-05-08T16:22:06.612 INFO:tasks.workunit.client.0.smithi172.stdout: "num_deep_scrub_errors": 0, 2024-05-08T16:22:06.612 INFO:tasks.workunit.client.0.smithi172.stdout: "num_objects_recovered": 0, 2024-05-08T16:22:06.612 INFO:tasks.workunit.client.0.smithi172.stdout: "num_bytes_recovered": 0, 2024-05-08T16:22:06.612 INFO:tasks.workunit.client.0.smithi172.stdout: "num_keys_recovered": 0, 2024-05-08T16:22:06.612 INFO:tasks.workunit.client.0.smithi172.stdout: "num_objects_omap": 0, 2024-05-08T16:22:06.612 INFO:tasks.workunit.client.0.smithi172.stdout: "num_objects_hit_set_archive": 0, 2024-05-08T16:22:06.612 INFO:tasks.workunit.client.0.smithi172.stdout: "num_bytes_hit_set_archive": 0, 2024-05-08T16:22:06.612 INFO:tasks.workunit.client.0.smithi172.stdout: "num_flush": 0, 2024-05-08T16:22:06.612 INFO:tasks.workunit.client.0.smithi172.stdout: "num_flush_kb": 0, 2024-05-08T16:22:06.612 INFO:tasks.workunit.client.0.smithi172.stdout: "num_evict": 0, 2024-05-08T16:22:06.613 INFO:tasks.workunit.client.0.smithi172.stdout: "num_evict_kb": 0, 2024-05-08T16:22:06.613 INFO:tasks.workunit.client.0.smithi172.stdout: "num_promote": 0, 2024-05-08T16:22:06.613 INFO:tasks.workunit.client.0.smithi172.stdout: "num_flush_mode_high": 0, 2024-05-08T16:22:06.613 INFO:tasks.workunit.client.0.smithi172.stdout: "num_flush_mode_low": 0, 2024-05-08T16:22:06.613 INFO:tasks.workunit.client.0.smithi172.stdout: "num_evict_mode_some": 0, 2024-05-08T16:22:06.614 INFO:tasks.workunit.client.0.smithi172.stdout: "num_evict_mode_full": 0, 2024-05-08T16:22:06.614 INFO:tasks.workunit.client.0.smithi172.stdout: "num_objects_pinned": 0, 2024-05-08T16:22:06.614 INFO:tasks.workunit.client.0.smithi172.stdout: "num_legacy_snapsets": 0, 2024-05-08T16:22:06.614 INFO:tasks.workunit.client.0.smithi172.stdout: "num_large_omap_objects": 0, 2024-05-08T16:22:06.614 INFO:tasks.workunit.client.0.smithi172.stdout: "num_objects_manifest": 0, 2024-05-08T16:22:06.614 INFO:tasks.workunit.client.0.smithi172.stdout: "num_omap_bytes": 0, 2024-05-08T16:22:06.614 INFO:tasks.workunit.client.0.smithi172.stdout: "num_omap_keys": 0, 2024-05-08T16:22:06.614 INFO:tasks.workunit.client.0.smithi172.stdout: "num_objects_repaired": 0 2024-05-08T16:22:06.614 INFO:tasks.workunit.client.0.smithi172.stdout: }, 2024-05-08T16:22:06.614 INFO:tasks.workunit.client.0.smithi172.stdout: "up": [ 2024-05-08T16:22:06.614 INFO:tasks.workunit.client.0.smithi172.stdout: 3, 2024-05-08T16:22:06.615 INFO:tasks.workunit.client.0.smithi172.stdout: 1, 2024-05-08T16:22:06.615 INFO:tasks.workunit.client.0.smithi172.stdout: 4, 2024-05-08T16:22:06.615 INFO:tasks.workunit.client.0.smithi172.stdout: 0, 2024-05-08T16:22:06.615 INFO:tasks.workunit.client.0.smithi172.stdout: 2147483647 2024-05-08T16:22:06.615 INFO:tasks.workunit.client.0.smithi172.stdout: ], 2024-05-08T16:22:06.615 INFO:tasks.workunit.client.0.smithi172.stdout: "acting": [ 2024-05-08T16:22:06.615 INFO:tasks.workunit.client.0.smithi172.stdout: 3, 2024-05-08T16:22:06.615 INFO:tasks.workunit.client.0.smithi172.stdout: 1, 2024-05-08T16:22:06.615 INFO:tasks.workunit.client.0.smithi172.stdout: 4, 2024-05-08T16:22:06.615 INFO:tasks.workunit.client.0.smithi172.stdout: 0, 2024-05-08T16:22:06.615 INFO:tasks.workunit.client.0.smithi172.stdout: 2147483647 2024-05-08T16:22:06.615 INFO:tasks.workunit.client.0.smithi172.stdout: ], 2024-05-08T16:22:06.615 INFO:tasks.workunit.client.0.smithi172.stdout: "avail_no_missing": [ 2024-05-08T16:22:06.615 INFO:tasks.workunit.client.0.smithi172.stdout: "3(0)", 2024-05-08T16:22:06.615 INFO:tasks.workunit.client.0.smithi172.stdout: "0(3)", 2024-05-08T16:22:06.616 INFO:tasks.workunit.client.0.smithi172.stdout: "1(1)", 2024-05-08T16:22:06.616 INFO:tasks.workunit.client.0.smithi172.stdout: "4(2)" 2024-05-08T16:22:06.616 INFO:tasks.workunit.client.0.smithi172.stdout: ], 2024-05-08T16:22:06.616 INFO:tasks.workunit.client.0.smithi172.stdout: "object_location_counts": [ 2024-05-08T16:22:06.616 INFO:tasks.workunit.client.0.smithi172.stdout: { 2024-05-08T16:22:06.616 INFO:tasks.workunit.client.0.smithi172.stdout: "shards": "0(3),1(1),3(0),4(2)", 2024-05-08T16:22:06.616 INFO:tasks.workunit.client.0.smithi172.stdout: "objects": 301 2024-05-08T16:22:06.616 INFO:tasks.workunit.client.0.smithi172.stdout: } 2024-05-08T16:22:06.616 INFO:tasks.workunit.client.0.smithi172.stdout: ], 2024-05-08T16:22:06.616 INFO:tasks.workunit.client.0.smithi172.stdout: "blocked_by": [], 2024-05-08T16:22:06.616 INFO:tasks.workunit.client.0.smithi172.stdout: "up_primary": 3, 2024-05-08T16:22:06.616 INFO:tasks.workunit.client.0.smithi172.stdout: "acting_primary": 3, 2024-05-08T16:22:06.616 INFO:tasks.workunit.client.0.smithi172.stdout: "purged_snaps": [] 2024-05-08T16:22:06.616 INFO:tasks.workunit.client.0.smithi172.stdout: }, 2024-05-08T16:22:06.616 INFO:tasks.workunit.client.0.smithi172.stdout: "empty": 0, 2024-05-08T16:22:06.616 INFO:tasks.workunit.client.0.smithi172.stdout: "dne": 0, 2024-05-08T16:22:06.617 INFO:tasks.workunit.client.0.smithi172.stdout: "incomplete": 0, 2024-05-08T16:22:06.617 INFO:tasks.workunit.client.0.smithi172.stdout: "last_epoch_started": 61, 2024-05-08T16:22:06.617 INFO:tasks.workunit.client.0.smithi172.stdout: "hit_set_history": { 2024-05-08T16:22:06.617 INFO:tasks.workunit.client.0.smithi172.stdout: "current_last_update": "0'0", 2024-05-08T16:22:06.617 INFO:tasks.workunit.client.0.smithi172.stdout: "history": [] 2024-05-08T16:22:06.617 INFO:tasks.workunit.client.0.smithi172.stdout: } 2024-05-08T16:22:06.617 INFO:tasks.workunit.client.0.smithi172.stdout: }, 2024-05-08T16:22:06.617 INFO:tasks.workunit.client.0.smithi172.stdout: { 2024-05-08T16:22:06.617 INFO:tasks.workunit.client.0.smithi172.stdout: "peer": "4(2)", 2024-05-08T16:22:06.617 INFO:tasks.workunit.client.0.smithi172.stdout: "pgid": "2.0s2", 2024-05-08T16:22:06.617 INFO:tasks.workunit.client.0.smithi172.stdout: "shared": "2", 2024-05-08T16:22:06.617 INFO:tasks.workunit.client.0.smithi172.stdout: "last_update": "50'301", 2024-05-08T16:22:06.617 INFO:tasks.workunit.client.0.smithi172.stdout: "last_complete": "50'301", 2024-05-08T16:22:06.617 INFO:tasks.workunit.client.0.smithi172.stdout: "log_tail": "50'200", 2024-05-08T16:22:06.617 INFO:tasks.workunit.client.0.smithi172.stdout: "last_user_version": 301, 2024-05-08T16:22:06.618 INFO:tasks.workunit.client.0.smithi172.stdout: "last_backfill": "MAX", 2024-05-08T16:22:06.618 INFO:tasks.workunit.client.0.smithi172.stdout: "purged_snaps": [], 2024-05-08T16:22:06.618 INFO:tasks.workunit.client.0.smithi172.stdout: "history": { 2024-05-08T16:22:06.618 INFO:tasks.workunit.client.0.smithi172.stdout: "epoch_created": 40, 2024-05-08T16:22:06.618 INFO:tasks.workunit.client.0.smithi172.stdout: "epoch_pool_created": 40, 2024-05-08T16:22:06.618 INFO:tasks.workunit.client.0.smithi172.stdout: "last_epoch_started": 61, 2024-05-08T16:22:06.618 INFO:tasks.workunit.client.0.smithi172.stdout: "last_interval_started": 60, 2024-05-08T16:22:06.618 INFO:tasks.workunit.client.0.smithi172.stdout: "last_epoch_clean": 43, 2024-05-08T16:22:06.618 INFO:tasks.workunit.client.0.smithi172.stdout: "last_interval_clean": 40, 2024-05-08T16:22:06.618 INFO:tasks.workunit.client.0.smithi172.stdout: "last_epoch_split": 0, 2024-05-08T16:22:06.618 INFO:tasks.workunit.client.0.smithi172.stdout: "last_epoch_marked_full": 0, 2024-05-08T16:22:06.618 INFO:tasks.workunit.client.0.smithi172.stdout: "same_up_since": 60, 2024-05-08T16:22:06.618 INFO:tasks.workunit.client.0.smithi172.stdout: "same_interval_since": 60, 2024-05-08T16:22:06.618 INFO:tasks.workunit.client.0.smithi172.stdout: "same_primary_since": 40, 2024-05-08T16:22:06.618 INFO:tasks.workunit.client.0.smithi172.stdout: "last_scrub": "0'0", 2024-05-08T16:22:06.619 INFO:tasks.workunit.client.0.smithi172.stdout: "last_scrub_stamp": "2024-05-08T16:19:38.829889+0000", 2024-05-08T16:22:06.619 INFO:tasks.workunit.client.0.smithi172.stdout: "last_deep_scrub": "0'0", 2024-05-08T16:22:06.619 INFO:tasks.workunit.client.0.smithi172.stdout: "last_deep_scrub_stamp": "2024-05-08T16:19:38.829889+0000", 2024-05-08T16:22:06.619 INFO:tasks.workunit.client.0.smithi172.stdout: "last_clean_scrub_stamp": "2024-05-08T16:19:38.829889+0000", 2024-05-08T16:22:06.619 INFO:tasks.workunit.client.0.smithi172.stdout: "prior_readable_until_ub": 0 2024-05-08T16:22:06.619 INFO:tasks.workunit.client.0.smithi172.stdout: }, 2024-05-08T16:22:06.619 INFO:tasks.workunit.client.0.smithi172.stdout: "stats": { 2024-05-08T16:22:06.619 INFO:tasks.workunit.client.0.smithi172.stdout: "version": "50'301", 2024-05-08T16:22:06.619 INFO:tasks.workunit.client.0.smithi172.stdout: "reported_seq": 336, 2024-05-08T16:22:06.619 INFO:tasks.workunit.client.0.smithi172.stdout: "reported_epoch": 50, 2024-05-08T16:22:06.619 INFO:tasks.workunit.client.0.smithi172.stdout: "state": "active+undersized+degraded", 2024-05-08T16:22:06.619 INFO:tasks.workunit.client.0.smithi172.stdout: "last_fresh": "2024-05-08T16:21:23.637408+0000", 2024-05-08T16:22:06.619 INFO:tasks.workunit.client.0.smithi172.stdout: "last_change": "2024-05-08T16:19:50.127512+0000", 2024-05-08T16:22:06.619 INFO:tasks.workunit.client.0.smithi172.stdout: "last_active": "2024-05-08T16:21:23.637408+0000", 2024-05-08T16:22:06.619 INFO:tasks.workunit.client.0.smithi172.stdout: "last_peered": "2024-05-08T16:21:23.637408+0000", 2024-05-08T16:22:06.620 INFO:tasks.workunit.client.0.smithi172.stdout: "last_clean": "2024-05-08T16:19:47.631736+0000", 2024-05-08T16:22:06.620 INFO:tasks.workunit.client.0.smithi172.stdout: "last_became_active": "2024-05-08T16:19:50.127512+0000", 2024-05-08T16:22:06.620 INFO:tasks.workunit.client.0.smithi172.stdout: "last_became_peered": "2024-05-08T16:19:50.127512+0000", 2024-05-08T16:22:06.620 INFO:tasks.workunit.client.0.smithi172.stdout: "last_unstale": "2024-05-08T16:21:23.637408+0000", 2024-05-08T16:22:06.620 INFO:tasks.workunit.client.0.smithi172.stdout: "last_undegraded": "2024-05-08T16:19:49.766134+0000", 2024-05-08T16:22:06.620 INFO:tasks.workunit.client.0.smithi172.stdout: "last_fullsized": "2024-05-08T16:19:49.765821+0000", 2024-05-08T16:22:06.620 INFO:tasks.workunit.client.0.smithi172.stdout: "mapping_epoch": 60, 2024-05-08T16:22:06.620 INFO:tasks.workunit.client.0.smithi172.stdout: "log_start": "50'200", 2024-05-08T16:22:06.620 INFO:tasks.workunit.client.0.smithi172.stdout: "ondisk_log_start": "50'200", 2024-05-08T16:22:06.620 INFO:tasks.workunit.client.0.smithi172.stdout: "created": 40, 2024-05-08T16:22:06.620 INFO:tasks.workunit.client.0.smithi172.stdout: "last_epoch_clean": 43, 2024-05-08T16:22:06.621 INFO:tasks.workunit.client.0.smithi172.stdout: "parent": "0.0", 2024-05-08T16:22:06.621 INFO:tasks.workunit.client.0.smithi172.stdout: "parent_split_bits": 0, 2024-05-08T16:22:06.621 INFO:tasks.workunit.client.0.smithi172.stdout: "last_scrub": "0'0", 2024-05-08T16:22:06.621 INFO:tasks.workunit.client.0.smithi172.stdout: "last_scrub_stamp": "2024-05-08T16:19:38.829889+0000", 2024-05-08T16:22:06.621 INFO:tasks.workunit.client.0.smithi172.stdout: "last_deep_scrub": "0'0", 2024-05-08T16:22:06.621 INFO:tasks.workunit.client.0.smithi172.stdout: "last_deep_scrub_stamp": "2024-05-08T16:19:38.829889+0000", 2024-05-08T16:22:06.621 INFO:tasks.workunit.client.0.smithi172.stdout: "last_clean_scrub_stamp": "2024-05-08T16:19:38.829889+0000", 2024-05-08T16:22:06.621 INFO:tasks.workunit.client.0.smithi172.stdout: "objects_scrubbed": 0, 2024-05-08T16:22:06.621 INFO:tasks.workunit.client.0.smithi172.stdout: "log_size": 101, 2024-05-08T16:22:06.621 INFO:tasks.workunit.client.0.smithi172.stdout: "log_dups_size": 200, 2024-05-08T16:22:06.621 INFO:tasks.workunit.client.0.smithi172.stdout: "ondisk_log_size": 101, 2024-05-08T16:22:06.621 INFO:tasks.workunit.client.0.smithi172.stdout: "stats_invalid": false, 2024-05-08T16:22:06.621 INFO:tasks.workunit.client.0.smithi172.stdout: "dirty_stats_invalid": false, 2024-05-08T16:22:06.621 INFO:tasks.workunit.client.0.smithi172.stdout: "omap_stats_invalid": false, 2024-05-08T16:22:06.621 INFO:tasks.workunit.client.0.smithi172.stdout: "hitset_stats_invalid": false, 2024-05-08T16:22:06.622 INFO:tasks.workunit.client.0.smithi172.stdout: "hitset_bytes_stats_invalid": false, 2024-05-08T16:22:06.622 INFO:tasks.workunit.client.0.smithi172.stdout: "pin_stats_invalid": false, 2024-05-08T16:22:06.622 INFO:tasks.workunit.client.0.smithi172.stdout: "manifest_stats_invalid": false, 2024-05-08T16:22:06.622 INFO:tasks.workunit.client.0.smithi172.stdout: "snaptrimq_len": 0, 2024-05-08T16:22:06.622 INFO:tasks.workunit.client.0.smithi172.stdout: "last_scrub_duration": 0, 2024-05-08T16:22:06.622 INFO:tasks.workunit.client.0.smithi172.stdout: "scrub_schedule": "periodic scrub scheduled @ 2024-05-09T17:43:21.349481+0000", 2024-05-08T16:22:06.622 INFO:tasks.workunit.client.0.smithi172.stdout: "scrub_duration": 0, 2024-05-08T16:22:06.622 INFO:tasks.workunit.client.0.smithi172.stdout: "objects_trimmed": 0, 2024-05-08T16:22:06.622 INFO:tasks.workunit.client.0.smithi172.stdout: "snaptrim_duration": 0, 2024-05-08T16:22:06.622 INFO:tasks.workunit.client.0.smithi172.stdout: "stat_sum": { 2024-05-08T16:22:06.622 INFO:tasks.workunit.client.0.smithi172.stdout: "num_bytes": 1232896, 2024-05-08T16:22:06.622 INFO:tasks.workunit.client.0.smithi172.stdout: "num_objects": 301, 2024-05-08T16:22:06.622 INFO:tasks.workunit.client.0.smithi172.stdout: "num_object_clones": 0, 2024-05-08T16:22:06.622 INFO:tasks.workunit.client.0.smithi172.stdout: "num_object_copies": 1505, 2024-05-08T16:22:06.623 INFO:tasks.workunit.client.0.smithi172.stdout: "num_objects_missing_on_primary": 0, 2024-05-08T16:22:06.623 INFO:tasks.workunit.client.0.smithi172.stdout: "num_objects_missing": 0, 2024-05-08T16:22:06.623 INFO:tasks.workunit.client.0.smithi172.stdout: "num_objects_degraded": 301, 2024-05-08T16:22:06.623 INFO:tasks.workunit.client.0.smithi172.stdout: "num_objects_misplaced": 0, 2024-05-08T16:22:06.623 INFO:tasks.workunit.client.0.smithi172.stdout: "num_objects_unfound": 0, 2024-05-08T16:22:06.623 INFO:tasks.workunit.client.0.smithi172.stdout: "num_objects_dirty": 301, 2024-05-08T16:22:06.623 INFO:tasks.workunit.client.0.smithi172.stdout: "num_whiteouts": 0, 2024-05-08T16:22:06.623 INFO:tasks.workunit.client.0.smithi172.stdout: "num_read": 0, 2024-05-08T16:22:06.623 INFO:tasks.workunit.client.0.smithi172.stdout: "num_read_kb": 0, 2024-05-08T16:22:06.623 INFO:tasks.workunit.client.0.smithi172.stdout: "num_write": 301, 2024-05-08T16:22:06.623 INFO:tasks.workunit.client.0.smithi172.stdout: "num_write_kb": 1204, 2024-05-08T16:22:06.623 INFO:tasks.workunit.client.0.smithi172.stdout: "num_scrub_errors": 0, 2024-05-08T16:22:06.623 INFO:tasks.workunit.client.0.smithi172.stdout: "num_shallow_scrub_errors": 0, 2024-05-08T16:22:06.623 INFO:tasks.workunit.client.0.smithi172.stdout: "num_deep_scrub_errors": 0, 2024-05-08T16:22:06.623 INFO:tasks.workunit.client.0.smithi172.stdout: "num_objects_recovered": 0, 2024-05-08T16:22:06.623 INFO:tasks.workunit.client.0.smithi172.stdout: "num_bytes_recovered": 0, 2024-05-08T16:22:06.624 INFO:tasks.workunit.client.0.smithi172.stdout: "num_keys_recovered": 0, 2024-05-08T16:22:06.624 INFO:tasks.workunit.client.0.smithi172.stdout: "num_objects_omap": 0, 2024-05-08T16:22:06.624 INFO:tasks.workunit.client.0.smithi172.stdout: "num_objects_hit_set_archive": 0, 2024-05-08T16:22:06.624 INFO:tasks.workunit.client.0.smithi172.stdout: "num_bytes_hit_set_archive": 0, 2024-05-08T16:22:06.624 INFO:tasks.workunit.client.0.smithi172.stdout: "num_flush": 0, 2024-05-08T16:22:06.624 INFO:tasks.workunit.client.0.smithi172.stdout: "num_flush_kb": 0, 2024-05-08T16:22:06.624 INFO:tasks.workunit.client.0.smithi172.stdout: "num_evict": 0, 2024-05-08T16:22:06.624 INFO:tasks.workunit.client.0.smithi172.stdout: "num_evict_kb": 0, 2024-05-08T16:22:06.624 INFO:tasks.workunit.client.0.smithi172.stdout: "num_promote": 0, 2024-05-08T16:22:06.624 INFO:tasks.workunit.client.0.smithi172.stdout: "num_flush_mode_high": 0, 2024-05-08T16:22:06.624 INFO:tasks.workunit.client.0.smithi172.stdout: "num_flush_mode_low": 0, 2024-05-08T16:22:06.624 INFO:tasks.workunit.client.0.smithi172.stdout: "num_evict_mode_some": 0, 2024-05-08T16:22:06.624 INFO:tasks.workunit.client.0.smithi172.stdout: "num_evict_mode_full": 0, 2024-05-08T16:22:06.624 INFO:tasks.workunit.client.0.smithi172.stdout: "num_objects_pinned": 0, 2024-05-08T16:22:06.624 INFO:tasks.workunit.client.0.smithi172.stdout: "num_legacy_snapsets": 0, 2024-05-08T16:22:06.625 INFO:tasks.workunit.client.0.smithi172.stdout: "num_large_omap_objects": 0, 2024-05-08T16:22:06.625 INFO:tasks.workunit.client.0.smithi172.stdout: "num_objects_manifest": 0, 2024-05-08T16:22:06.625 INFO:tasks.workunit.client.0.smithi172.stdout: "num_omap_bytes": 0, 2024-05-08T16:22:06.625 INFO:tasks.workunit.client.0.smithi172.stdout: "num_omap_keys": 0, 2024-05-08T16:22:06.625 INFO:tasks.workunit.client.0.smithi172.stdout: "num_objects_repaired": 0 2024-05-08T16:22:06.625 INFO:tasks.workunit.client.0.smithi172.stdout: }, 2024-05-08T16:22:06.625 INFO:tasks.workunit.client.0.smithi172.stdout: "up": [ 2024-05-08T16:22:06.625 INFO:tasks.workunit.client.0.smithi172.stdout: 3, 2024-05-08T16:22:06.625 INFO:tasks.workunit.client.0.smithi172.stdout: 1, 2024-05-08T16:22:06.625 INFO:tasks.workunit.client.0.smithi172.stdout: 4, 2024-05-08T16:22:06.625 INFO:tasks.workunit.client.0.smithi172.stdout: 0, 2024-05-08T16:22:06.625 INFO:tasks.workunit.client.0.smithi172.stdout: 2147483647 2024-05-08T16:22:06.625 INFO:tasks.workunit.client.0.smithi172.stdout: ], 2024-05-08T16:22:06.625 INFO:tasks.workunit.client.0.smithi172.stdout: "acting": [ 2024-05-08T16:22:06.625 INFO:tasks.workunit.client.0.smithi172.stdout: 3, 2024-05-08T16:22:06.625 INFO:tasks.workunit.client.0.smithi172.stdout: 1, 2024-05-08T16:22:06.626 INFO:tasks.workunit.client.0.smithi172.stdout: 4, 2024-05-08T16:22:06.626 INFO:tasks.workunit.client.0.smithi172.stdout: 0, 2024-05-08T16:22:06.626 INFO:tasks.workunit.client.0.smithi172.stdout: 2147483647 2024-05-08T16:22:06.626 INFO:tasks.workunit.client.0.smithi172.stdout: ], 2024-05-08T16:22:06.626 INFO:tasks.workunit.client.0.smithi172.stdout: "avail_no_missing": [ 2024-05-08T16:22:06.626 INFO:tasks.workunit.client.0.smithi172.stdout: "3(0)", 2024-05-08T16:22:06.626 INFO:tasks.workunit.client.0.smithi172.stdout: "0(3)", 2024-05-08T16:22:06.626 INFO:tasks.workunit.client.0.smithi172.stdout: "1(1)", 2024-05-08T16:22:06.626 INFO:tasks.workunit.client.0.smithi172.stdout: "4(2)" 2024-05-08T16:22:06.626 INFO:tasks.workunit.client.0.smithi172.stdout: ], 2024-05-08T16:22:06.626 INFO:tasks.workunit.client.0.smithi172.stdout: "object_location_counts": [ 2024-05-08T16:22:06.626 INFO:tasks.workunit.client.0.smithi172.stdout: { 2024-05-08T16:22:06.626 INFO:tasks.workunit.client.0.smithi172.stdout: "shards": "0(3),1(1),3(0),4(2)", 2024-05-08T16:22:06.626 INFO:tasks.workunit.client.0.smithi172.stdout: "objects": 301 2024-05-08T16:22:06.626 INFO:tasks.workunit.client.0.smithi172.stdout: } 2024-05-08T16:22:06.627 INFO:tasks.workunit.client.0.smithi172.stdout: ], 2024-05-08T16:22:06.627 INFO:tasks.workunit.client.0.smithi172.stdout: "blocked_by": [], 2024-05-08T16:22:06.627 INFO:tasks.workunit.client.0.smithi172.stdout: "up_primary": 3, 2024-05-08T16:22:06.627 INFO:tasks.workunit.client.0.smithi172.stdout: "acting_primary": 3, 2024-05-08T16:22:06.627 INFO:tasks.workunit.client.0.smithi172.stdout: "purged_snaps": [] 2024-05-08T16:22:06.627 INFO:tasks.workunit.client.0.smithi172.stdout: }, 2024-05-08T16:22:06.627 INFO:tasks.workunit.client.0.smithi172.stdout: "empty": 0, 2024-05-08T16:22:06.627 INFO:tasks.workunit.client.0.smithi172.stdout: "dne": 0, 2024-05-08T16:22:06.627 INFO:tasks.workunit.client.0.smithi172.stdout: "incomplete": 0, 2024-05-08T16:22:06.627 INFO:tasks.workunit.client.0.smithi172.stdout: "last_epoch_started": 61, 2024-05-08T16:22:06.627 INFO:tasks.workunit.client.0.smithi172.stdout: "hit_set_history": { 2024-05-08T16:22:06.627 INFO:tasks.workunit.client.0.smithi172.stdout: "current_last_update": "0'0", 2024-05-08T16:22:06.627 INFO:tasks.workunit.client.0.smithi172.stdout: "history": [] 2024-05-08T16:22:06.627 INFO:tasks.workunit.client.0.smithi172.stdout: } 2024-05-08T16:22:06.627 INFO:tasks.workunit.client.0.smithi172.stdout: } 2024-05-08T16:22:06.628 INFO:tasks.workunit.client.0.smithi172.stdout: ], 2024-05-08T16:22:06.628 INFO:tasks.workunit.client.0.smithi172.stdout: "recovery_state": [ 2024-05-08T16:22:06.628 INFO:tasks.workunit.client.0.smithi172.stdout: { 2024-05-08T16:22:06.628 INFO:tasks.workunit.client.0.smithi172.stdout: "name": "Started/Primary/Active", 2024-05-08T16:22:06.628 INFO:tasks.workunit.client.0.smithi172.stdout: "enter_time": "2024-05-08T16:22:01.521512+0000", 2024-05-08T16:22:06.628 INFO:tasks.workunit.client.0.smithi172.stdout: "might_have_unfound": [ 2024-05-08T16:22:06.628 INFO:tasks.workunit.client.0.smithi172.stdout: { 2024-05-08T16:22:06.628 INFO:tasks.workunit.client.0.smithi172.stdout: "osd": "0(3)", 2024-05-08T16:22:06.628 INFO:tasks.workunit.client.0.smithi172.stdout: "status": "already probed" 2024-05-08T16:22:06.628 INFO:tasks.workunit.client.0.smithi172.stdout: }, 2024-05-08T16:22:06.628 INFO:tasks.workunit.client.0.smithi172.stdout: { 2024-05-08T16:22:06.628 INFO:tasks.workunit.client.0.smithi172.stdout: "osd": "1(1)", 2024-05-08T16:22:06.628 INFO:tasks.workunit.client.0.smithi172.stdout: "status": "already probed" 2024-05-08T16:22:06.628 INFO:tasks.workunit.client.0.smithi172.stdout: }, 2024-05-08T16:22:06.628 INFO:tasks.workunit.client.0.smithi172.stdout: { 2024-05-08T16:22:06.629 INFO:tasks.workunit.client.0.smithi172.stdout: "osd": "2(4)", 2024-05-08T16:22:06.629 INFO:tasks.workunit.client.0.smithi172.stdout: "status": "osd is down" 2024-05-08T16:22:06.629 INFO:tasks.workunit.client.0.smithi172.stdout: }, 2024-05-08T16:22:06.629 INFO:tasks.workunit.client.0.smithi172.stdout: { 2024-05-08T16:22:06.629 INFO:tasks.workunit.client.0.smithi172.stdout: "osd": "4(2)", 2024-05-08T16:22:06.629 INFO:tasks.workunit.client.0.smithi172.stdout: "status": "already probed" 2024-05-08T16:22:06.629 INFO:tasks.workunit.client.0.smithi172.stdout: } 2024-05-08T16:22:06.629 INFO:tasks.workunit.client.0.smithi172.stdout: ], 2024-05-08T16:22:06.629 INFO:tasks.workunit.client.0.smithi172.stdout: "recovery_progress": { 2024-05-08T16:22:06.629 INFO:tasks.workunit.client.0.smithi172.stdout: "backfill_targets": [], 2024-05-08T16:22:06.629 INFO:tasks.workunit.client.0.smithi172.stdout: "waiting_on_backfill": [], 2024-05-08T16:22:06.629 INFO:tasks.workunit.client.0.smithi172.stdout: "last_backfill_started": "MIN", 2024-05-08T16:22:06.629 INFO:tasks.workunit.client.0.smithi172.stdout: "backfill_info": { 2024-05-08T16:22:06.629 INFO:tasks.workunit.client.0.smithi172.stdout: "begin": "MIN", 2024-05-08T16:22:06.629 INFO:tasks.workunit.client.0.smithi172.stdout: "end": "MIN", 2024-05-08T16:22:06.629 INFO:tasks.workunit.client.0.smithi172.stdout: "objects": [] 2024-05-08T16:22:06.630 INFO:tasks.workunit.client.0.smithi172.stdout: }, 2024-05-08T16:22:06.630 INFO:tasks.workunit.client.0.smithi172.stdout: "peer_backfill_info": [], 2024-05-08T16:22:06.630 INFO:tasks.workunit.client.0.smithi172.stdout: "backfills_in_flight": [], 2024-05-08T16:22:06.630 INFO:tasks.workunit.client.0.smithi172.stdout: "recovering": [], 2024-05-08T16:22:06.630 INFO:tasks.workunit.client.0.smithi172.stdout: "pg_backend": { 2024-05-08T16:22:06.630 INFO:tasks.workunit.client.0.smithi172.stdout: "recovery_ops": [], 2024-05-08T16:22:06.630 INFO:tasks.workunit.client.0.smithi172.stdout: "read_ops": [] 2024-05-08T16:22:06.630 INFO:tasks.workunit.client.0.smithi172.stdout: } 2024-05-08T16:22:06.630 INFO:tasks.workunit.client.0.smithi172.stdout: } 2024-05-08T16:22:06.630 INFO:tasks.workunit.client.0.smithi172.stdout: }, 2024-05-08T16:22:06.630 INFO:tasks.workunit.client.0.smithi172.stdout: { 2024-05-08T16:22:06.630 INFO:tasks.workunit.client.0.smithi172.stdout: "name": "Started", 2024-05-08T16:22:06.630 INFO:tasks.workunit.client.0.smithi172.stdout: "enter_time": "2024-05-08T16:22:00.986825+0000" 2024-05-08T16:22:06.630 INFO:tasks.workunit.client.0.smithi172.stdout: } 2024-05-08T16:22:06.630 INFO:tasks.workunit.client.0.smithi172.stdout: ], 2024-05-08T16:22:06.631 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubber": { 2024-05-08T16:22:06.631 INFO:tasks.workunit.client.0.smithi172.stdout: "active": false, 2024-05-08T16:22:06.631 INFO:tasks.workunit.client.0.smithi172.stdout: "must_scrub": false, 2024-05-08T16:22:06.631 INFO:tasks.workunit.client.0.smithi172.stdout: "must_deep_scrub": false, 2024-05-08T16:22:06.631 INFO:tasks.workunit.client.0.smithi172.stdout: "must_repair": false, 2024-05-08T16:22:06.631 INFO:tasks.workunit.client.0.smithi172.stdout: "need_auto": false, 2024-05-08T16:22:06.631 INFO:tasks.workunit.client.0.smithi172.stdout: "scrub_reg_stamp": "2024-05-09T17:43:21.349481+0000", 2024-05-08T16:22:06.631 INFO:tasks.workunit.client.0.smithi172.stdout: "schedule": "no scrub is scheduled" 2024-05-08T16:22:06.631 INFO:tasks.workunit.client.0.smithi172.stdout: }, 2024-05-08T16:22:06.631 INFO:tasks.workunit.client.0.smithi172.stdout: "agent_state": {} 2024-05-08T16:22:06.631 INFO:tasks.workunit.client.0.smithi172.stdout:} 2024-05-08T16:22:06.632 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:570: TEST_ec_backfill_unfound: ceph pg 2.0 list_unfound 2024-05-08T16:22:06.632 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:570: TEST_ec_backfill_unfound: grep -q obj250 2024-05-08T16:22:06.750 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:572: TEST_ec_backfill_unfound: ceph pg 2.0 list_unfound 2024-05-08T16:22:06.750 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:572: TEST_ec_backfill_unfound: jq .available_might_have_unfound 2024-05-08T16:22:06.891 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:572: TEST_ec_backfill_unfound: check=true 2024-05-08T16:22:06.891 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:573: TEST_ec_backfill_unfound: test true == true 2024-05-08T16:22:06.891 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:575: TEST_ec_backfill_unfound: ceph pg 2.0 list_unfound 2024-05-08T16:22:06.891 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:575: TEST_ec_backfill_unfound: jq '.might_have_unfound[0].status' 2024-05-08T16:22:07.035 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:575: TEST_ec_backfill_unfound: eval 'check="osd' is 'down"' 2024-05-08T16:22:07.036 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:575: TEST_ec_backfill_unfound: check='osd is down' 2024-05-08T16:22:07.036 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:576: TEST_ec_backfill_unfound: test 'osd is down' == 'osd is down' 2024-05-08T16:22:07.036 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: ceph pg 2.0 list_unfound 2024-05-08T16:22:07.036 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: jq '.might_have_unfound[0].osd' 2024-05-08T16:22:07.196 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: eval 'check="2(4)"' 2024-05-08T16:22:07.197 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:578: TEST_ec_backfill_unfound: check='2(4)' 2024-05-08T16:22:07.197 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:579: TEST_ec_backfill_unfound: test '2(4)' == '2(4)' 2024-05-08T16:22:07.197 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:581: TEST_ec_backfill_unfound: activate_osd td/test-erasure-eio 2 2024-05-08T16:22:07.197 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/test-erasure-eio 2024-05-08T16:22:07.197 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-05-08T16:22:07.197 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-05-08T16:22:07.197 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-05-08T16:22:07.197 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/test-erasure-eio/2 2024-05-08T16:22:07.197 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=4457b0f1-028b-482b-8ee3-7ee05861e6cc --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-05-08T16:22:07.197 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-08T16:22:07.197 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-05-08T16:22:07.198 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-08T16:22:07.198 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/2' 2024-05-08T16:22:07.198 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/2/journal' 2024-05-08T16:22:07.198 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-05-08T16:22:07.198 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-05-08T16:22:07.198 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-05-08T16:22:07.198 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-05-08T16:22:07.198 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-08T16:22:07.198 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-08T16:22:07.198 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-08T16:22:07.198 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T16:22:07.198 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81110 2024-05-08T16:22:07.199 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.81110/$cluster-$name.asok' 2024-05-08T16:22:07.199 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.81110/$cluster-$name.asok' 2024-05-08T16:22:07.199 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-05-08T16:22:07.199 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-05-08T16:22:07.199 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-05-08T16:22:07.199 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-08T16:22:07.199 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-08T16:22:07.199 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-08T16:22:07.199 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-08T16:22:07.199 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-05-08T16:22:07.199 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-05-08T16:22:07.200 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/test-erasure-eio/2 2024-05-08T16:22:07.200 INFO:tasks.workunit.client.0.smithi172.stdout:start osd.2 2024-05-08T16:22:07.201 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-05-08T16:22:07.201 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=4457b0f1-028b-482b-8ee3-7ee05861e6cc --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81110/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-05-08T16:22:07.201 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/test-erasure-eio/2/whoami 2024-05-08T16:22:07.201 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-05-08T16:22:07.201 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-05-08T16:22:07.201 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-05-08T16:22:07.202 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-05-08T16:22:07.223 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:22:07.228+0000 7fd4546dc7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:22:07.226 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:22:07.228+0000 7fd4546dc7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:22:07.228 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:22:07.232+0000 7fd4546dc7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:22:07.593 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-05-08T16:22:07.593 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-08T16:22:07.593 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-05-08T16:22:07.594 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-08T16:22:07.594 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-08T16:22:07.594 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:22:07.594 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-08T16:22:07.594 INFO:tasks.workunit.client.0.smithi172.stdout:0 2024-05-08T16:22:07.594 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:22:07.594 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T16:22:07.801 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:22:07.804+0000 7fd4546dc7c0 -1 Falling back to public interface 2024-05-08T16:22:08.023 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:22:09.024 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:22:09.024 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:22:09.025 INFO:tasks.workunit.client.0.smithi172.stdout:1 2024-05-08T16:22:09.025 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-08T16:22:09.025 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:22:09.026 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T16:22:09.441 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:22:10.442 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:22:10.443 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:22:10.443 INFO:tasks.workunit.client.0.smithi172.stdout:2 2024-05-08T16:22:10.443 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-08T16:22:10.443 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:22:10.443 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T16:22:10.863 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:22:10.889 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:22:10.892+0000 7fd4546dc7c0 -1 osd.2 59 log_to_monitors true 2024-05-08T16:22:11.864 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:22:11.865 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:22:11.865 INFO:tasks.workunit.client.0.smithi172.stdout:3 2024-05-08T16:22:11.865 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-05-08T16:22:11.865 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:22:11.865 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T16:22:12.387 INFO:tasks.workunit.client.0.smithi172.stdout:osd.2 up in weight 1 up_from 63 up_thru 20 down_at 60 last_clean_interval [52,59) [v2:127.0.0.1:6818/2218280482,v1:127.0.0.1:6819/2218280482] [v2:127.0.0.1:6820/2218280482,v1:127.0.0.1:6821/2218280482] exists,up 91b6118c-0348-4a21-beef-0db38b0f7cce 2024-05-08T16:22:12.387 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-08T16:22:12.387 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-08T16:22:12.387 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-08T16:22:12.387 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:584: TEST_ec_backfill_unfound: timeout 5 rados -p pool-jerasure get obj250 td/test-erasure-eio/CHECK 2024-05-08T16:22:17.390 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:585: TEST_ec_backfill_unfound: test 124 = 124 2024-05-08T16:22:17.391 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:587: TEST_ec_backfill_unfound: ceph pg 2.0 mark_unfound_lost delete 2024-05-08T16:22:17.683 INFO:tasks.workunit.client.0.smithi172.stderr:pg has no unfound objects 2024-05-08T16:22:17.704 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:589: TEST_ec_backfill_unfound: wait_for_clean 2024-05-08T16:22:17.704 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-05-08T16:22:17.704 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-05-08T16:22:17.704 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-05-08T16:22:17.705 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-05-08T16:22:17.705 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-05-08T16:22:17.705 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-05-08T16:22:17.705 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-05-08T16:22:17.706 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-05-08T16:22:17.706 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-05-08T16:22:17.815 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-05-08T16:22:17.815 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-05-08T16:22:17.815 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-05-08T16:22:17.815 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-05-08T16:22:17.815 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-05-08T16:22:17.815 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-05-08T16:22:18.229 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-05-08T16:22:18.229 INFO:tasks.workunit.client.0.smithi172.stderr:1 2024-05-08T16:22:18.230 INFO:tasks.workunit.client.0.smithi172.stderr:2 2024-05-08T16:22:18.230 INFO:tasks.workunit.client.0.smithi172.stderr:3 2024-05-08T16:22:18.230 INFO:tasks.workunit.client.0.smithi172.stderr:4' 2024-05-08T16:22:18.230 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-05-08T16:22:18.230 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T16:22:18.230 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-05-08T16:22:18.437 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=25769803850 2024-05-08T16:22:18.437 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 25769803850 2024-05-08T16:22:18.437 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803850' 2024-05-08T16:22:18.437 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T16:22:18.437 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-05-08T16:22:18.597 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=55834574909 2024-05-08T16:22:18.597 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 55834574909 2024-05-08T16:22:18.597 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803850 1-55834574909' 2024-05-08T16:22:18.597 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T16:22:18.597 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-05-08T16:22:18.831 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=270582939652 2024-05-08T16:22:18.831 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 270582939652 2024-05-08T16:22:18.831 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803850 1-55834574909 2-270582939652' 2024-05-08T16:22:18.831 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T16:22:18.831 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-05-08T16:22:18.987 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=115964117041 2024-05-08T16:22:18.987 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 115964117041 2024-05-08T16:22:18.987 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803850 1-55834574909 2-270582939652 3-115964117041' 2024-05-08T16:22:18.987 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T16:22:18.987 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-05-08T16:22:19.144 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=154618822699 2024-05-08T16:22:19.144 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 154618822699 2024-05-08T16:22:19.144 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803850 1-55834574909 2-270582939652 3-115964117041 4-154618822699' 2024-05-08T16:22:19.144 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-08T16:22:19.145 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-25769803850 2024-05-08T16:22:19.145 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-08T16:22:19.145 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-05-08T16:22:19.146 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-25769803850 2024-05-08T16:22:19.146 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-08T16:22:19.147 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=25769803850 2024-05-08T16:22:19.147 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 25769803850' 2024-05-08T16:22:19.147 INFO:tasks.workunit.client.0.smithi172.stdout:waiting osd.0 seq 25769803850 2024-05-08T16:22:19.148 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-05-08T16:22:19.550 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 25769803848 -lt 25769803850 2024-05-08T16:22:19.550 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-05-08T16:22:20.552 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-05-08T16:22:20.552 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-05-08T16:22:21.042 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 25769803850 -lt 25769803850 2024-05-08T16:22:21.042 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-08T16:22:21.043 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-55834574909 2024-05-08T16:22:21.044 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-08T16:22:21.044 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-05-08T16:22:21.045 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-55834574909 2024-05-08T16:22:21.045 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-08T16:22:21.047 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=55834574909 2024-05-08T16:22:21.047 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 55834574909' 2024-05-08T16:22:21.048 INFO:tasks.workunit.client.0.smithi172.stdout:waiting osd.1 seq 55834574909 2024-05-08T16:22:21.048 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-05-08T16:22:21.661 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 55834574909 -lt 55834574909 2024-05-08T16:22:21.662 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-08T16:22:21.662 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-270582939652 2024-05-08T16:22:21.663 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-08T16:22:21.664 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-05-08T16:22:21.665 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-08T16:22:21.665 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-270582939652 2024-05-08T16:22:21.666 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=270582939652 2024-05-08T16:22:21.666 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 270582939652' 2024-05-08T16:22:21.667 INFO:tasks.workunit.client.0.smithi172.stdout:waiting osd.2 seq 270582939652 2024-05-08T16:22:21.667 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-05-08T16:22:22.361 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 270582939653 -lt 270582939652 2024-05-08T16:22:22.362 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-08T16:22:22.362 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-115964117041 2024-05-08T16:22:22.362 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-08T16:22:22.364 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-05-08T16:22:22.364 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-115964117041 2024-05-08T16:22:22.364 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-08T16:22:22.365 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=115964117041 2024-05-08T16:22:22.365 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 115964117041' 2024-05-08T16:22:22.366 INFO:tasks.workunit.client.0.smithi172.stdout:waiting osd.3 seq 115964117041 2024-05-08T16:22:22.366 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-05-08T16:22:22.854 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 115964117041 -lt 115964117041 2024-05-08T16:22:22.854 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-08T16:22:22.855 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-154618822699 2024-05-08T16:22:22.855 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-08T16:22:22.856 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-05-08T16:22:22.857 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-154618822699 2024-05-08T16:22:22.857 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-08T16:22:22.858 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=154618822699 2024-05-08T16:22:22.858 INFO:tasks.workunit.client.0.smithi172.stdout:waiting osd.4 seq 154618822699 2024-05-08T16:22:22.859 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 154618822699' 2024-05-08T16:22:22.859 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-05-08T16:22:23.294 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 154618822699 -lt 154618822699 2024-05-08T16:22:23.294 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-05-08T16:22:23.295 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-05-08T16:22:23.295 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-05-08T16:22:23.917 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-05-08T16:22:23.917 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-05-08T16:22:23.918 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-05-08T16:22:23.918 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-05-08T16:22:23.918 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-05-08T16:22:23.918 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-05-08T16:22:23.918 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-05-08T16:22:23.918 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-05-08T16:22:24.397 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-05-08T16:22:24.398 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-05-08T16:22:24.398 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-05-08T16:22:24.398 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-05-08T16:22:24.938 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 5 2024-05-08T16:22:24.938 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' -1 2024-05-08T16:22:24.938 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-05-08T16:22:24.938 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=4 2024-05-08T16:22:24.938 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-05-08T16:22:24.938 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-05-08T16:22:25.038 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-05-08T16:22:25.039 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-05-08T16:22:25.039 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-05-08T16:22:25.039 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-05-08T16:22:25.039 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-05-08T16:22:25.040 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-05-08T16:22:25.040 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-05-08T16:22:25.040 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-05-08T16:22:25.448 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-05-08T16:22:25.448 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-05-08T16:22:25.449 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-05-08T16:22:25.449 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-05-08T16:22:26.003 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 5 2024-05-08T16:22:26.003 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' 4 2024-05-08T16:22:26.003 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-05-08T16:22:26.003 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-05-08T16:22:26.003 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-05-08T16:22:26.003 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-05-08T16:22:26.004 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-05-08T16:22:26.004 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-05-08T16:22:26.004 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-05-08T16:22:26.525 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-05-08T16:22:26.525 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-05-08T16:22:26.525 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2024-05-08T16:22:26.526 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-05-08T16:22:26.526 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-05-08T16:22:26.726 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-05-08T16:22:26.727 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-05-08T16:22:26.727 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-05-08T16:22:26.727 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-05-08T16:22:26.728 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-05-08T16:22:26.728 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-05-08T16:22:26.728 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-05-08T16:22:26.728 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-05-08T16:22:27.114 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-05-08T16:22:27.114 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-05-08T16:22:27.114 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-05-08T16:22:27.114 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-05-08T16:22:27.768 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-05-08T16:22:27.768 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-05-08T16:22:27.769 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-05-08T16:22:27.769 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: seq 1 300 2024-05-08T16:22:27.770 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:22:27.770 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj1 = obj250 ']' 2024-05-08T16:22:27.770 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj1 td/test-erasure-eio/CHECK 2024-05-08T16:22:27.839 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:22:27.840 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:22:27.840 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj2 = obj250 ']' 2024-05-08T16:22:27.840 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj2 td/test-erasure-eio/CHECK 2024-05-08T16:22:27.871 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:22:27.873 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:22:27.873 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj3 = obj250 ']' 2024-05-08T16:22:27.873 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj3 td/test-erasure-eio/CHECK 2024-05-08T16:22:27.923 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:22:27.924 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:22:27.924 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj4 = obj250 ']' 2024-05-08T16:22:27.924 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj4 td/test-erasure-eio/CHECK 2024-05-08T16:22:27.955 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:22:27.956 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:22:27.956 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj5 = obj250 ']' 2024-05-08T16:22:27.956 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj5 td/test-erasure-eio/CHECK 2024-05-08T16:22:27.985 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:22:27.986 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:22:27.986 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj6 = obj250 ']' 2024-05-08T16:22:27.986 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj6 td/test-erasure-eio/CHECK 2024-05-08T16:22:28.015 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:22:28.016 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:22:28.016 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj7 = obj250 ']' 2024-05-08T16:22:28.016 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj7 td/test-erasure-eio/CHECK 2024-05-08T16:22:28.045 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:22:28.046 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:22:28.046 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj8 = obj250 ']' 2024-05-08T16:22:28.047 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj8 td/test-erasure-eio/CHECK 2024-05-08T16:22:28.075 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:22:28.076 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:22:28.076 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj9 = obj250 ']' 2024-05-08T16:22:28.076 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj9 td/test-erasure-eio/CHECK 2024-05-08T16:22:28.104 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:22:28.105 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:22:28.106 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj10 = obj250 ']' 2024-05-08T16:22:28.106 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj10 td/test-erasure-eio/CHECK 2024-05-08T16:22:28.172 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:22:28.173 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:22:28.173 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj11 = obj250 ']' 2024-05-08T16:22:28.173 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj11 td/test-erasure-eio/CHECK 2024-05-08T16:22:28.201 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:22:28.202 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:22:28.202 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj12 = obj250 ']' 2024-05-08T16:22:28.202 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj12 td/test-erasure-eio/CHECK 2024-05-08T16:22:28.257 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:22:28.258 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:22:28.258 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj13 = obj250 ']' 2024-05-08T16:22:28.258 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj13 td/test-erasure-eio/CHECK 2024-05-08T16:22:28.288 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:22:28.288 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:22:28.288 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj14 = obj250 ']' 2024-05-08T16:22:28.289 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj14 td/test-erasure-eio/CHECK 2024-05-08T16:22:28.316 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:22:28.317 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:22:28.317 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj15 = obj250 ']' 2024-05-08T16:22:28.317 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj15 td/test-erasure-eio/CHECK 2024-05-08T16:22:28.346 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:22:28.347 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:22:28.347 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj16 = obj250 ']' 2024-05-08T16:22:28.347 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj16 td/test-erasure-eio/CHECK 2024-05-08T16:22:28.376 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:22:28.377 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:22:28.377 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj17 = obj250 ']' 2024-05-08T16:22:28.377 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj17 td/test-erasure-eio/CHECK 2024-05-08T16:22:28.406 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:22:28.407 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:22:28.407 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj18 = obj250 ']' 2024-05-08T16:22:28.407 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj18 td/test-erasure-eio/CHECK 2024-05-08T16:22:28.435 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:22:28.436 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:22:28.436 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj19 = obj250 ']' 2024-05-08T16:22:28.436 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj19 td/test-erasure-eio/CHECK 2024-05-08T16:22:28.464 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:22:28.465 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:22:28.465 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj20 = obj250 ']' 2024-05-08T16:22:28.465 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj20 td/test-erasure-eio/CHECK 2024-05-08T16:22:28.493 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:22:28.494 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:22:28.494 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj21 = obj250 ']' 2024-05-08T16:22:28.494 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj21 td/test-erasure-eio/CHECK 2024-05-08T16:22:28.522 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:22:28.523 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:22:28.523 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj22 = obj250 ']' 2024-05-08T16:22:28.523 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj22 td/test-erasure-eio/CHECK 2024-05-08T16:22:28.553 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:22:28.554 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:22:28.554 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj23 = obj250 ']' 2024-05-08T16:22:28.554 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj23 td/test-erasure-eio/CHECK 2024-05-08T16:22:28.584 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:22:28.584 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:22:28.585 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj24 = obj250 ']' 2024-05-08T16:22:28.585 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj24 td/test-erasure-eio/CHECK 2024-05-08T16:22:28.614 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:22:28.615 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:22:28.615 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj25 = obj250 ']' 2024-05-08T16:22:28.615 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj25 td/test-erasure-eio/CHECK 2024-05-08T16:22:28.644 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:22:28.644 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:22:28.644 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj26 = obj250 ']' 2024-05-08T16:22:28.645 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj26 td/test-erasure-eio/CHECK 2024-05-08T16:22:28.673 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:22:28.673 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:22:28.674 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj27 = obj250 ']' 2024-05-08T16:22:28.674 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj27 td/test-erasure-eio/CHECK 2024-05-08T16:22:28.702 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:22:28.702 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:22:28.703 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj28 = obj250 ']' 2024-05-08T16:22:28.703 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj28 td/test-erasure-eio/CHECK 2024-05-08T16:22:28.731 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:22:28.732 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:22:28.732 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj29 = obj250 ']' 2024-05-08T16:22:28.732 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj29 td/test-erasure-eio/CHECK 2024-05-08T16:22:28.760 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:22:28.761 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:22:28.761 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj30 = obj250 ']' 2024-05-08T16:22:28.761 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj30 td/test-erasure-eio/CHECK 2024-05-08T16:22:28.789 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:22:28.790 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:22:28.790 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj31 = obj250 ']' 2024-05-08T16:22:28.790 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj31 td/test-erasure-eio/CHECK 2024-05-08T16:22:28.819 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:22:28.820 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:22:28.820 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj32 = obj250 ']' 2024-05-08T16:22:28.820 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj32 td/test-erasure-eio/CHECK 2024-05-08T16:22:28.849 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:22:28.849 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:22:28.849 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj33 = obj250 ']' 2024-05-08T16:22:28.850 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj33 td/test-erasure-eio/CHECK 2024-05-08T16:22:28.891 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:22:28.892 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:22:28.892 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj34 = obj250 ']' 2024-05-08T16:22:28.892 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj34 td/test-erasure-eio/CHECK 2024-05-08T16:22:28.921 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:22:28.921 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:22:28.921 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj35 = obj250 ']' 2024-05-08T16:22:28.922 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj35 td/test-erasure-eio/CHECK 2024-05-08T16:22:28.950 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:22:28.951 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:22:28.951 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj36 = obj250 ']' 2024-05-08T16:22:28.951 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj36 td/test-erasure-eio/CHECK 2024-05-08T16:22:28.979 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:22:28.980 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:22:28.980 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj37 = obj250 ']' 2024-05-08T16:22:28.980 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj37 td/test-erasure-eio/CHECK 2024-05-08T16:22:29.009 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:22:29.010 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:22:29.010 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj38 = obj250 ']' 2024-05-08T16:22:29.010 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj38 td/test-erasure-eio/CHECK 2024-05-08T16:22:29.038 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:22:29.039 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:22:29.039 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj39 = obj250 ']' 2024-05-08T16:22:29.039 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj39 td/test-erasure-eio/CHECK 2024-05-08T16:22:29.067 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:22:29.068 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:22:29.068 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj40 = obj250 ']' 2024-05-08T16:22:29.068 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj40 td/test-erasure-eio/CHECK 2024-05-08T16:22:29.096 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:22:29.097 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:22:29.097 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj41 = obj250 ']' 2024-05-08T16:22:29.097 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj41 td/test-erasure-eio/CHECK 2024-05-08T16:22:29.126 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:22:29.127 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:22:29.127 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj42 = obj250 ']' 2024-05-08T16:22:29.127 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj42 td/test-erasure-eio/CHECK 2024-05-08T16:22:29.155 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:22:29.156 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:22:29.156 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj43 = obj250 ']' 2024-05-08T16:22:29.156 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj43 td/test-erasure-eio/CHECK 2024-05-08T16:22:29.184 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:22:29.185 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:22:29.185 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj44 = obj250 ']' 2024-05-08T16:22:29.185 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj44 td/test-erasure-eio/CHECK 2024-05-08T16:22:29.213 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:22:29.214 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:22:29.214 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj45 = obj250 ']' 2024-05-08T16:22:29.214 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj45 td/test-erasure-eio/CHECK 2024-05-08T16:22:29.242 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:22:29.243 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:22:29.243 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj46 = obj250 ']' 2024-05-08T16:22:29.243 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj46 td/test-erasure-eio/CHECK 2024-05-08T16:22:29.271 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:22:29.272 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:22:29.272 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj47 = obj250 ']' 2024-05-08T16:22:29.272 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj47 td/test-erasure-eio/CHECK 2024-05-08T16:22:29.300 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:22:29.301 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:22:29.301 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj48 = obj250 ']' 2024-05-08T16:22:29.301 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj48 td/test-erasure-eio/CHECK 2024-05-08T16:22:29.329 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:22:29.330 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:22:29.330 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj49 = obj250 ']' 2024-05-08T16:22:29.330 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj49 td/test-erasure-eio/CHECK 2024-05-08T16:22:29.358 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:22:29.359 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:22:29.360 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj50 = obj250 ']' 2024-05-08T16:22:29.360 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj50 td/test-erasure-eio/CHECK 2024-05-08T16:22:29.389 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:22:29.389 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:22:29.389 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj51 = obj250 ']' 2024-05-08T16:22:29.389 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj51 td/test-erasure-eio/CHECK 2024-05-08T16:22:29.417 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:22:29.418 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:22:29.418 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj52 = obj250 ']' 2024-05-08T16:22:29.418 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj52 td/test-erasure-eio/CHECK 2024-05-08T16:22:29.446 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:22:29.447 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:22:29.447 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj53 = obj250 ']' 2024-05-08T16:22:29.447 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj53 td/test-erasure-eio/CHECK 2024-05-08T16:22:29.475 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:22:29.476 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:22:29.476 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj54 = obj250 ']' 2024-05-08T16:22:29.476 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj54 td/test-erasure-eio/CHECK 2024-05-08T16:22:29.505 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:22:29.505 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:22:29.505 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj55 = obj250 ']' 2024-05-08T16:22:29.506 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj55 td/test-erasure-eio/CHECK 2024-05-08T16:22:29.534 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:22:29.535 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:22:29.535 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj56 = obj250 ']' 2024-05-08T16:22:29.535 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj56 td/test-erasure-eio/CHECK 2024-05-08T16:22:29.563 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:22:29.564 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:22:29.564 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj57 = obj250 ']' 2024-05-08T16:22:29.564 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj57 td/test-erasure-eio/CHECK 2024-05-08T16:22:29.592 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:22:29.593 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:22:29.593 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj58 = obj250 ']' 2024-05-08T16:22:29.593 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj58 td/test-erasure-eio/CHECK 2024-05-08T16:22:29.621 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:22:29.622 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:22:29.622 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj59 = obj250 ']' 2024-05-08T16:22:29.622 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj59 td/test-erasure-eio/CHECK 2024-05-08T16:22:29.650 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:22:29.651 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:22:29.651 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj60 = obj250 ']' 2024-05-08T16:22:29.651 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj60 td/test-erasure-eio/CHECK 2024-05-08T16:22:29.680 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:22:29.680 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:22:29.681 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj61 = obj250 ']' 2024-05-08T16:22:29.681 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj61 td/test-erasure-eio/CHECK 2024-05-08T16:22:29.739 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:22:29.740 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:22:29.740 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj62 = obj250 ']' 2024-05-08T16:22:29.740 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj62 td/test-erasure-eio/CHECK 2024-05-08T16:22:29.768 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:22:29.770 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:22:29.770 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj63 = obj250 ']' 2024-05-08T16:22:29.770 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj63 td/test-erasure-eio/CHECK 2024-05-08T16:22:29.797 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:22:29.798 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:22:29.798 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj64 = obj250 ']' 2024-05-08T16:22:29.798 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj64 td/test-erasure-eio/CHECK 2024-05-08T16:22:29.826 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:22:29.827 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:22:29.827 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj65 = obj250 ']' 2024-05-08T16:22:29.827 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj65 td/test-erasure-eio/CHECK 2024-05-08T16:22:29.855 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:22:29.856 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:22:29.856 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj66 = obj250 ']' 2024-05-08T16:22:29.856 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj66 td/test-erasure-eio/CHECK 2024-05-08T16:22:29.884 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:22:29.885 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:22:29.885 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj67 = obj250 ']' 2024-05-08T16:22:29.885 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj67 td/test-erasure-eio/CHECK 2024-05-08T16:22:29.913 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:22:29.914 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:22:29.914 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj68 = obj250 ']' 2024-05-08T16:22:29.914 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj68 td/test-erasure-eio/CHECK 2024-05-08T16:22:29.942 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:22:29.943 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:22:29.943 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj69 = obj250 ']' 2024-05-08T16:22:29.943 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj69 td/test-erasure-eio/CHECK 2024-05-08T16:22:29.972 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:22:29.972 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:22:29.973 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj70 = obj250 ']' 2024-05-08T16:22:29.973 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj70 td/test-erasure-eio/CHECK 2024-05-08T16:22:30.001 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:22:30.002 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:22:30.002 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj71 = obj250 ']' 2024-05-08T16:22:30.002 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj71 td/test-erasure-eio/CHECK 2024-05-08T16:22:30.030 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:22:30.031 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:22:30.031 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj72 = obj250 ']' 2024-05-08T16:22:30.031 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj72 td/test-erasure-eio/CHECK 2024-05-08T16:22:30.059 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:22:30.060 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:22:30.060 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj73 = obj250 ']' 2024-05-08T16:22:30.060 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj73 td/test-erasure-eio/CHECK 2024-05-08T16:22:30.088 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:22:30.089 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:22:30.089 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj74 = obj250 ']' 2024-05-08T16:22:30.089 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj74 td/test-erasure-eio/CHECK 2024-05-08T16:22:30.117 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:22:30.118 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:22:30.118 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj75 = obj250 ']' 2024-05-08T16:22:30.118 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj75 td/test-erasure-eio/CHECK 2024-05-08T16:22:30.164 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:22:30.165 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:22:30.165 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj76 = obj250 ']' 2024-05-08T16:22:30.165 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj76 td/test-erasure-eio/CHECK 2024-05-08T16:22:30.194 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:22:30.195 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:22:30.195 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj77 = obj250 ']' 2024-05-08T16:22:30.195 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj77 td/test-erasure-eio/CHECK 2024-05-08T16:22:30.266 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:22:30.267 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:22:30.267 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj78 = obj250 ']' 2024-05-08T16:22:30.267 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj78 td/test-erasure-eio/CHECK 2024-05-08T16:22:30.296 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:22:30.297 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:22:30.297 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj79 = obj250 ']' 2024-05-08T16:22:30.297 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj79 td/test-erasure-eio/CHECK 2024-05-08T16:22:30.325 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:22:30.326 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:22:30.326 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj80 = obj250 ']' 2024-05-08T16:22:30.326 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj80 td/test-erasure-eio/CHECK 2024-05-08T16:22:30.354 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:22:30.355 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:22:30.355 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj81 = obj250 ']' 2024-05-08T16:22:30.355 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj81 td/test-erasure-eio/CHECK 2024-05-08T16:22:30.383 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:22:30.384 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:22:30.384 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj82 = obj250 ']' 2024-05-08T16:22:30.384 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj82 td/test-erasure-eio/CHECK 2024-05-08T16:22:30.412 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:22:30.413 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:22:30.413 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj83 = obj250 ']' 2024-05-08T16:22:30.413 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj83 td/test-erasure-eio/CHECK 2024-05-08T16:22:30.441 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:22:30.442 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:22:30.442 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj84 = obj250 ']' 2024-05-08T16:22:30.442 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj84 td/test-erasure-eio/CHECK 2024-05-08T16:22:30.470 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:22:30.471 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:22:30.471 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj85 = obj250 ']' 2024-05-08T16:22:30.471 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj85 td/test-erasure-eio/CHECK 2024-05-08T16:22:30.499 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:22:30.500 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:22:30.500 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj86 = obj250 ']' 2024-05-08T16:22:30.500 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj86 td/test-erasure-eio/CHECK 2024-05-08T16:22:30.528 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:22:30.529 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:22:30.529 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj87 = obj250 ']' 2024-05-08T16:22:30.529 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj87 td/test-erasure-eio/CHECK 2024-05-08T16:22:30.557 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:22:30.558 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:22:30.558 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj88 = obj250 ']' 2024-05-08T16:22:30.558 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj88 td/test-erasure-eio/CHECK 2024-05-08T16:22:30.586 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:22:30.587 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:22:30.587 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj89 = obj250 ']' 2024-05-08T16:22:30.587 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj89 td/test-erasure-eio/CHECK 2024-05-08T16:22:30.615 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:22:30.616 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:22:30.616 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj90 = obj250 ']' 2024-05-08T16:22:30.617 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj90 td/test-erasure-eio/CHECK 2024-05-08T16:22:30.644 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:22:30.645 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:22:30.645 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj91 = obj250 ']' 2024-05-08T16:22:30.645 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj91 td/test-erasure-eio/CHECK 2024-05-08T16:22:30.673 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:22:30.674 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:22:30.674 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj92 = obj250 ']' 2024-05-08T16:22:30.674 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj92 td/test-erasure-eio/CHECK 2024-05-08T16:22:30.702 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:22:30.703 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:22:30.703 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj93 = obj250 ']' 2024-05-08T16:22:30.703 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj93 td/test-erasure-eio/CHECK 2024-05-08T16:22:30.731 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:22:30.732 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:22:30.732 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj94 = obj250 ']' 2024-05-08T16:22:30.732 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj94 td/test-erasure-eio/CHECK 2024-05-08T16:22:30.760 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:22:30.761 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:22:30.761 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj95 = obj250 ']' 2024-05-08T16:22:30.761 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj95 td/test-erasure-eio/CHECK 2024-05-08T16:22:30.789 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:22:30.790 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:22:30.790 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj96 = obj250 ']' 2024-05-08T16:22:30.790 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj96 td/test-erasure-eio/CHECK 2024-05-08T16:22:30.819 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:22:30.820 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:22:30.820 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj97 = obj250 ']' 2024-05-08T16:22:30.820 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj97 td/test-erasure-eio/CHECK 2024-05-08T16:22:30.847 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:22:30.848 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:22:30.848 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj98 = obj250 ']' 2024-05-08T16:22:30.848 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj98 td/test-erasure-eio/CHECK 2024-05-08T16:22:30.877 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:22:30.878 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:22:30.878 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj99 = obj250 ']' 2024-05-08T16:22:30.878 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj99 td/test-erasure-eio/CHECK 2024-05-08T16:22:30.906 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:22:30.907 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:22:30.907 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj100 = obj250 ']' 2024-05-08T16:22:30.907 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj100 td/test-erasure-eio/CHECK 2024-05-08T16:22:30.935 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:22:30.935 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:22:30.936 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj101 = obj250 ']' 2024-05-08T16:22:30.936 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj101 td/test-erasure-eio/CHECK 2024-05-08T16:22:30.964 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:22:30.965 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:22:30.965 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj102 = obj250 ']' 2024-05-08T16:22:30.965 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj102 td/test-erasure-eio/CHECK 2024-05-08T16:22:30.993 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:22:30.994 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:22:30.994 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj103 = obj250 ']' 2024-05-08T16:22:30.994 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj103 td/test-erasure-eio/CHECK 2024-05-08T16:22:31.022 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:22:31.023 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:22:31.023 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj104 = obj250 ']' 2024-05-08T16:22:31.023 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj104 td/test-erasure-eio/CHECK 2024-05-08T16:22:31.051 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:22:31.052 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:22:31.052 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj105 = obj250 ']' 2024-05-08T16:22:31.052 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj105 td/test-erasure-eio/CHECK 2024-05-08T16:22:31.081 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:22:31.081 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:22:31.082 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj106 = obj250 ']' 2024-05-08T16:22:31.082 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj106 td/test-erasure-eio/CHECK 2024-05-08T16:22:31.110 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:22:31.111 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:22:31.111 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj107 = obj250 ']' 2024-05-08T16:22:31.111 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj107 td/test-erasure-eio/CHECK 2024-05-08T16:22:31.139 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:22:31.140 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:22:31.140 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj108 = obj250 ']' 2024-05-08T16:22:31.140 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj108 td/test-erasure-eio/CHECK 2024-05-08T16:22:31.168 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:22:31.169 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:22:31.169 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj109 = obj250 ']' 2024-05-08T16:22:31.169 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj109 td/test-erasure-eio/CHECK 2024-05-08T16:22:31.197 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:22:31.198 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:22:31.199 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj110 = obj250 ']' 2024-05-08T16:22:31.199 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj110 td/test-erasure-eio/CHECK 2024-05-08T16:22:31.226 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:22:31.227 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:22:31.227 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj111 = obj250 ']' 2024-05-08T16:22:31.228 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj111 td/test-erasure-eio/CHECK 2024-05-08T16:22:31.256 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:22:31.256 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:22:31.257 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj112 = obj250 ']' 2024-05-08T16:22:31.257 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj112 td/test-erasure-eio/CHECK 2024-05-08T16:22:31.285 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:22:31.286 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:22:31.286 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj113 = obj250 ']' 2024-05-08T16:22:31.286 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj113 td/test-erasure-eio/CHECK 2024-05-08T16:22:31.315 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:22:31.316 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:22:31.316 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj114 = obj250 ']' 2024-05-08T16:22:31.316 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj114 td/test-erasure-eio/CHECK 2024-05-08T16:22:31.344 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:22:31.345 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:22:31.345 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj115 = obj250 ']' 2024-05-08T16:22:31.345 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj115 td/test-erasure-eio/CHECK 2024-05-08T16:22:31.373 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:22:31.374 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:22:31.375 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj116 = obj250 ']' 2024-05-08T16:22:31.375 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj116 td/test-erasure-eio/CHECK 2024-05-08T16:22:31.403 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:22:31.404 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:22:31.404 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj117 = obj250 ']' 2024-05-08T16:22:31.404 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj117 td/test-erasure-eio/CHECK 2024-05-08T16:22:31.432 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:22:31.433 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:22:31.433 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj118 = obj250 ']' 2024-05-08T16:22:31.433 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj118 td/test-erasure-eio/CHECK 2024-05-08T16:22:31.462 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:22:31.463 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:22:31.463 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj119 = obj250 ']' 2024-05-08T16:22:31.463 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj119 td/test-erasure-eio/CHECK 2024-05-08T16:22:31.491 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:22:31.492 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:22:31.492 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj120 = obj250 ']' 2024-05-08T16:22:31.492 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj120 td/test-erasure-eio/CHECK 2024-05-08T16:22:31.520 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:22:31.521 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:22:31.521 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj121 = obj250 ']' 2024-05-08T16:22:31.521 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj121 td/test-erasure-eio/CHECK 2024-05-08T16:22:31.549 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:22:31.550 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:22:31.550 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj122 = obj250 ']' 2024-05-08T16:22:31.550 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj122 td/test-erasure-eio/CHECK 2024-05-08T16:22:31.578 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:22:31.579 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:22:31.579 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj123 = obj250 ']' 2024-05-08T16:22:31.579 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj123 td/test-erasure-eio/CHECK 2024-05-08T16:22:31.607 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:22:31.608 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:22:31.608 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj124 = obj250 ']' 2024-05-08T16:22:31.608 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj124 td/test-erasure-eio/CHECK 2024-05-08T16:22:31.636 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:22:31.637 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:22:31.637 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj125 = obj250 ']' 2024-05-08T16:22:31.637 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj125 td/test-erasure-eio/CHECK 2024-05-08T16:22:31.666 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:22:31.667 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:22:31.667 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj126 = obj250 ']' 2024-05-08T16:22:31.667 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj126 td/test-erasure-eio/CHECK 2024-05-08T16:22:31.730 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:22:31.731 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:22:31.731 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj127 = obj250 ']' 2024-05-08T16:22:31.731 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj127 td/test-erasure-eio/CHECK 2024-05-08T16:22:31.759 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:22:31.760 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:22:31.761 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj128 = obj250 ']' 2024-05-08T16:22:31.761 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj128 td/test-erasure-eio/CHECK 2024-05-08T16:22:31.789 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:22:31.790 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:22:31.790 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj129 = obj250 ']' 2024-05-08T16:22:31.790 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj129 td/test-erasure-eio/CHECK 2024-05-08T16:22:31.818 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:22:31.819 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:22:31.819 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj130 = obj250 ']' 2024-05-08T16:22:31.819 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj130 td/test-erasure-eio/CHECK 2024-05-08T16:22:31.847 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:22:31.848 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:22:31.848 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj131 = obj250 ']' 2024-05-08T16:22:31.848 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj131 td/test-erasure-eio/CHECK 2024-05-08T16:22:31.876 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:22:31.877 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:22:31.877 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj132 = obj250 ']' 2024-05-08T16:22:31.877 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj132 td/test-erasure-eio/CHECK 2024-05-08T16:22:31.905 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:22:31.906 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:22:31.906 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj133 = obj250 ']' 2024-05-08T16:22:31.906 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj133 td/test-erasure-eio/CHECK 2024-05-08T16:22:31.934 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:22:31.935 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:22:31.935 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj134 = obj250 ']' 2024-05-08T16:22:31.935 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj134 td/test-erasure-eio/CHECK 2024-05-08T16:22:31.963 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:22:31.964 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:22:31.964 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj135 = obj250 ']' 2024-05-08T16:22:31.964 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj135 td/test-erasure-eio/CHECK 2024-05-08T16:22:31.992 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:22:31.993 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:22:31.993 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj136 = obj250 ']' 2024-05-08T16:22:31.993 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj136 td/test-erasure-eio/CHECK 2024-05-08T16:22:32.021 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:22:32.022 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:22:32.022 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj137 = obj250 ']' 2024-05-08T16:22:32.022 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj137 td/test-erasure-eio/CHECK 2024-05-08T16:22:32.050 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:22:32.051 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:22:32.051 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj138 = obj250 ']' 2024-05-08T16:22:32.051 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj138 td/test-erasure-eio/CHECK 2024-05-08T16:22:32.080 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:22:32.081 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:22:32.081 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj139 = obj250 ']' 2024-05-08T16:22:32.081 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj139 td/test-erasure-eio/CHECK 2024-05-08T16:22:32.109 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:22:32.110 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:22:32.110 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj140 = obj250 ']' 2024-05-08T16:22:32.110 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj140 td/test-erasure-eio/CHECK 2024-05-08T16:22:32.165 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:22:32.166 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:22:32.166 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj141 = obj250 ']' 2024-05-08T16:22:32.166 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj141 td/test-erasure-eio/CHECK 2024-05-08T16:22:32.194 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:22:32.195 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:22:32.195 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj142 = obj250 ']' 2024-05-08T16:22:32.195 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj142 td/test-erasure-eio/CHECK 2024-05-08T16:22:32.278 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:22:32.279 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:22:32.279 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj143 = obj250 ']' 2024-05-08T16:22:32.279 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj143 td/test-erasure-eio/CHECK 2024-05-08T16:22:32.312 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:22:32.313 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:22:32.313 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj144 = obj250 ']' 2024-05-08T16:22:32.313 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj144 td/test-erasure-eio/CHECK 2024-05-08T16:22:32.341 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:22:32.342 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:22:32.342 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj145 = obj250 ']' 2024-05-08T16:22:32.343 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj145 td/test-erasure-eio/CHECK 2024-05-08T16:22:32.370 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:22:32.371 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:22:32.371 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj146 = obj250 ']' 2024-05-08T16:22:32.371 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj146 td/test-erasure-eio/CHECK 2024-05-08T16:22:32.399 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:22:32.400 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:22:32.400 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj147 = obj250 ']' 2024-05-08T16:22:32.401 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj147 td/test-erasure-eio/CHECK 2024-05-08T16:22:32.428 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:22:32.429 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:22:32.429 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj148 = obj250 ']' 2024-05-08T16:22:32.429 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj148 td/test-erasure-eio/CHECK 2024-05-08T16:22:32.457 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:22:32.458 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:22:32.458 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj149 = obj250 ']' 2024-05-08T16:22:32.458 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj149 td/test-erasure-eio/CHECK 2024-05-08T16:22:32.486 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:22:32.488 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:22:32.488 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj150 = obj250 ']' 2024-05-08T16:22:32.488 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj150 td/test-erasure-eio/CHECK 2024-05-08T16:22:32.516 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:22:32.517 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:22:32.517 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj151 = obj250 ']' 2024-05-08T16:22:32.517 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj151 td/test-erasure-eio/CHECK 2024-05-08T16:22:32.546 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:22:32.546 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:22:32.546 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj152 = obj250 ']' 2024-05-08T16:22:32.546 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj152 td/test-erasure-eio/CHECK 2024-05-08T16:22:32.575 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:22:32.575 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:22:32.576 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj153 = obj250 ']' 2024-05-08T16:22:32.576 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj153 td/test-erasure-eio/CHECK 2024-05-08T16:22:32.604 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:22:32.605 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:22:32.605 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj154 = obj250 ']' 2024-05-08T16:22:32.605 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj154 td/test-erasure-eio/CHECK 2024-05-08T16:22:32.633 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:22:32.635 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:22:32.635 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj155 = obj250 ']' 2024-05-08T16:22:32.635 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj155 td/test-erasure-eio/CHECK 2024-05-08T16:22:32.662 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:22:32.664 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:22:32.664 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj156 = obj250 ']' 2024-05-08T16:22:32.664 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj156 td/test-erasure-eio/CHECK 2024-05-08T16:22:32.691 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:22:32.692 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:22:32.692 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj157 = obj250 ']' 2024-05-08T16:22:32.692 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj157 td/test-erasure-eio/CHECK 2024-05-08T16:22:32.720 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:22:32.721 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:22:32.722 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj158 = obj250 ']' 2024-05-08T16:22:32.722 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj158 td/test-erasure-eio/CHECK 2024-05-08T16:22:32.749 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:22:32.750 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:22:32.750 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj159 = obj250 ']' 2024-05-08T16:22:32.750 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj159 td/test-erasure-eio/CHECK 2024-05-08T16:22:32.778 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:22:32.779 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:22:32.779 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj160 = obj250 ']' 2024-05-08T16:22:32.779 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj160 td/test-erasure-eio/CHECK 2024-05-08T16:22:32.807 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:22:32.809 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:22:32.809 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj161 = obj250 ']' 2024-05-08T16:22:32.809 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj161 td/test-erasure-eio/CHECK 2024-05-08T16:22:32.837 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:22:32.838 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:22:32.838 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj162 = obj250 ']' 2024-05-08T16:22:32.838 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj162 td/test-erasure-eio/CHECK 2024-05-08T16:22:32.866 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:22:32.867 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:22:32.867 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj163 = obj250 ']' 2024-05-08T16:22:32.867 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj163 td/test-erasure-eio/CHECK 2024-05-08T16:22:32.895 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:22:32.896 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:22:32.896 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj164 = obj250 ']' 2024-05-08T16:22:32.896 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj164 td/test-erasure-eio/CHECK 2024-05-08T16:22:32.924 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:22:32.925 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:22:32.925 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj165 = obj250 ']' 2024-05-08T16:22:32.925 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj165 td/test-erasure-eio/CHECK 2024-05-08T16:22:32.953 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:22:32.954 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:22:32.954 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj166 = obj250 ']' 2024-05-08T16:22:32.954 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj166 td/test-erasure-eio/CHECK 2024-05-08T16:22:32.982 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:22:32.983 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:22:32.983 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj167 = obj250 ']' 2024-05-08T16:22:32.983 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj167 td/test-erasure-eio/CHECK 2024-05-08T16:22:33.011 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:22:33.012 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:22:33.012 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj168 = obj250 ']' 2024-05-08T16:22:33.012 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj168 td/test-erasure-eio/CHECK 2024-05-08T16:22:33.040 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:22:33.040 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:22:33.041 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj169 = obj250 ']' 2024-05-08T16:22:33.041 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj169 td/test-erasure-eio/CHECK 2024-05-08T16:22:33.069 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:22:33.070 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:22:33.070 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj170 = obj250 ']' 2024-05-08T16:22:33.070 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj170 td/test-erasure-eio/CHECK 2024-05-08T16:22:33.098 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:22:33.099 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:22:33.099 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj171 = obj250 ']' 2024-05-08T16:22:33.099 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj171 td/test-erasure-eio/CHECK 2024-05-08T16:22:33.127 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:22:33.128 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:22:33.128 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj172 = obj250 ']' 2024-05-08T16:22:33.128 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj172 td/test-erasure-eio/CHECK 2024-05-08T16:22:33.158 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:22:33.159 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:22:33.159 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj173 = obj250 ']' 2024-05-08T16:22:33.159 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj173 td/test-erasure-eio/CHECK 2024-05-08T16:22:33.185 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:22:33.186 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:22:33.186 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj174 = obj250 ']' 2024-05-08T16:22:33.186 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj174 td/test-erasure-eio/CHECK 2024-05-08T16:22:33.214 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:22:33.215 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:22:33.215 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj175 = obj250 ']' 2024-05-08T16:22:33.215 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj175 td/test-erasure-eio/CHECK 2024-05-08T16:22:33.243 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:22:33.244 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:22:33.244 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj176 = obj250 ']' 2024-05-08T16:22:33.244 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj176 td/test-erasure-eio/CHECK 2024-05-08T16:22:33.272 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:22:33.273 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:22:33.273 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj177 = obj250 ']' 2024-05-08T16:22:33.273 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj177 td/test-erasure-eio/CHECK 2024-05-08T16:22:33.302 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:22:33.303 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:22:33.303 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj178 = obj250 ']' 2024-05-08T16:22:33.303 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj178 td/test-erasure-eio/CHECK 2024-05-08T16:22:33.331 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:22:33.332 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:22:33.332 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj179 = obj250 ']' 2024-05-08T16:22:33.332 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj179 td/test-erasure-eio/CHECK 2024-05-08T16:22:33.360 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:22:33.361 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:22:33.361 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj180 = obj250 ']' 2024-05-08T16:22:33.361 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj180 td/test-erasure-eio/CHECK 2024-05-08T16:22:33.389 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:22:33.390 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:22:33.390 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj181 = obj250 ']' 2024-05-08T16:22:33.390 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj181 td/test-erasure-eio/CHECK 2024-05-08T16:22:33.418 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:22:33.419 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:22:33.419 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj182 = obj250 ']' 2024-05-08T16:22:33.419 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj182 td/test-erasure-eio/CHECK 2024-05-08T16:22:33.448 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:22:33.449 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:22:33.449 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj183 = obj250 ']' 2024-05-08T16:22:33.449 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj183 td/test-erasure-eio/CHECK 2024-05-08T16:22:33.477 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:22:33.478 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:22:33.478 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj184 = obj250 ']' 2024-05-08T16:22:33.478 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj184 td/test-erasure-eio/CHECK 2024-05-08T16:22:33.507 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:22:33.508 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:22:33.508 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj185 = obj250 ']' 2024-05-08T16:22:33.508 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj185 td/test-erasure-eio/CHECK 2024-05-08T16:22:33.536 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:22:33.537 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:22:33.537 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj186 = obj250 ']' 2024-05-08T16:22:33.537 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj186 td/test-erasure-eio/CHECK 2024-05-08T16:22:33.565 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:22:33.566 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:22:33.566 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj187 = obj250 ']' 2024-05-08T16:22:33.566 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj187 td/test-erasure-eio/CHECK 2024-05-08T16:22:33.594 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:22:33.595 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:22:33.596 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj188 = obj250 ']' 2024-05-08T16:22:33.596 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj188 td/test-erasure-eio/CHECK 2024-05-08T16:22:33.623 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:22:33.624 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:22:33.624 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj189 = obj250 ']' 2024-05-08T16:22:33.624 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj189 td/test-erasure-eio/CHECK 2024-05-08T16:22:33.653 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:22:33.653 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:22:33.653 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj190 = obj250 ']' 2024-05-08T16:22:33.653 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj190 td/test-erasure-eio/CHECK 2024-05-08T16:22:33.682 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:22:33.683 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:22:33.683 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj191 = obj250 ']' 2024-05-08T16:22:33.683 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj191 td/test-erasure-eio/CHECK 2024-05-08T16:22:33.750 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:22:33.751 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:22:33.751 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj192 = obj250 ']' 2024-05-08T16:22:33.751 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj192 td/test-erasure-eio/CHECK 2024-05-08T16:22:33.779 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:22:33.780 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:22:33.780 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj193 = obj250 ']' 2024-05-08T16:22:33.780 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj193 td/test-erasure-eio/CHECK 2024-05-08T16:22:33.809 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:22:33.810 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:22:33.810 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj194 = obj250 ']' 2024-05-08T16:22:33.810 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj194 td/test-erasure-eio/CHECK 2024-05-08T16:22:33.838 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:22:33.839 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:22:33.839 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj195 = obj250 ']' 2024-05-08T16:22:33.839 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj195 td/test-erasure-eio/CHECK 2024-05-08T16:22:33.867 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:22:33.868 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:22:33.868 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj196 = obj250 ']' 2024-05-08T16:22:33.868 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj196 td/test-erasure-eio/CHECK 2024-05-08T16:22:33.896 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:22:33.897 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:22:33.897 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj197 = obj250 ']' 2024-05-08T16:22:33.897 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj197 td/test-erasure-eio/CHECK 2024-05-08T16:22:33.926 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:22:33.926 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:22:33.926 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj198 = obj250 ']' 2024-05-08T16:22:33.927 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj198 td/test-erasure-eio/CHECK 2024-05-08T16:22:33.955 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:22:33.956 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:22:33.956 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj199 = obj250 ']' 2024-05-08T16:22:33.956 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj199 td/test-erasure-eio/CHECK 2024-05-08T16:22:33.985 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:22:33.985 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:22:33.985 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj200 = obj250 ']' 2024-05-08T16:22:33.986 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj200 td/test-erasure-eio/CHECK 2024-05-08T16:22:34.014 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:22:34.015 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:22:34.015 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj201 = obj250 ']' 2024-05-08T16:22:34.015 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj201 td/test-erasure-eio/CHECK 2024-05-08T16:22:34.043 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:22:34.043 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:22:34.044 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj202 = obj250 ']' 2024-05-08T16:22:34.044 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj202 td/test-erasure-eio/CHECK 2024-05-08T16:22:34.072 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:22:34.072 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:22:34.072 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj203 = obj250 ']' 2024-05-08T16:22:34.073 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj203 td/test-erasure-eio/CHECK 2024-05-08T16:22:34.101 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:22:34.102 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:22:34.102 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj204 = obj250 ']' 2024-05-08T16:22:34.102 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj204 td/test-erasure-eio/CHECK 2024-05-08T16:22:34.167 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:22:34.167 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:22:34.168 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj205 = obj250 ']' 2024-05-08T16:22:34.168 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj205 td/test-erasure-eio/CHECK 2024-05-08T16:22:34.198 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:22:34.198 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:22:34.198 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj206 = obj250 ']' 2024-05-08T16:22:34.199 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj206 td/test-erasure-eio/CHECK 2024-05-08T16:22:34.252 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:22:34.253 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:22:34.253 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj207 = obj250 ']' 2024-05-08T16:22:34.253 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj207 td/test-erasure-eio/CHECK 2024-05-08T16:22:34.281 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:22:34.282 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:22:34.282 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj208 = obj250 ']' 2024-05-08T16:22:34.282 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj208 td/test-erasure-eio/CHECK 2024-05-08T16:22:34.310 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:22:34.311 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:22:34.311 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj209 = obj250 ']' 2024-05-08T16:22:34.311 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj209 td/test-erasure-eio/CHECK 2024-05-08T16:22:34.339 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:22:34.339 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:22:34.340 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj210 = obj250 ']' 2024-05-08T16:22:34.340 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj210 td/test-erasure-eio/CHECK 2024-05-08T16:22:34.368 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:22:34.369 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:22:34.369 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj211 = obj250 ']' 2024-05-08T16:22:34.369 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj211 td/test-erasure-eio/CHECK 2024-05-08T16:22:34.397 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:22:34.398 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:22:34.398 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj212 = obj250 ']' 2024-05-08T16:22:34.398 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj212 td/test-erasure-eio/CHECK 2024-05-08T16:22:34.426 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:22:34.427 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:22:34.427 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj213 = obj250 ']' 2024-05-08T16:22:34.427 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj213 td/test-erasure-eio/CHECK 2024-05-08T16:22:34.455 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:22:34.456 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:22:34.456 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj214 = obj250 ']' 2024-05-08T16:22:34.456 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj214 td/test-erasure-eio/CHECK 2024-05-08T16:22:34.484 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:22:34.485 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:22:34.485 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj215 = obj250 ']' 2024-05-08T16:22:34.485 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj215 td/test-erasure-eio/CHECK 2024-05-08T16:22:34.514 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:22:34.514 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:22:34.515 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj216 = obj250 ']' 2024-05-08T16:22:34.515 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj216 td/test-erasure-eio/CHECK 2024-05-08T16:22:34.543 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:22:34.544 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:22:34.544 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj217 = obj250 ']' 2024-05-08T16:22:34.544 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj217 td/test-erasure-eio/CHECK 2024-05-08T16:22:34.572 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:22:34.573 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:22:34.573 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj218 = obj250 ']' 2024-05-08T16:22:34.573 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj218 td/test-erasure-eio/CHECK 2024-05-08T16:22:34.601 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:22:34.602 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:22:34.602 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj219 = obj250 ']' 2024-05-08T16:22:34.602 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj219 td/test-erasure-eio/CHECK 2024-05-08T16:22:34.631 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:22:34.632 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:22:34.632 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj220 = obj250 ']' 2024-05-08T16:22:34.632 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj220 td/test-erasure-eio/CHECK 2024-05-08T16:22:34.661 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:22:34.661 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:22:34.661 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj221 = obj250 ']' 2024-05-08T16:22:34.661 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj221 td/test-erasure-eio/CHECK 2024-05-08T16:22:34.689 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:22:34.690 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:22:34.690 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj222 = obj250 ']' 2024-05-08T16:22:34.690 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj222 td/test-erasure-eio/CHECK 2024-05-08T16:22:34.718 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:22:34.719 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:22:34.719 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj223 = obj250 ']' 2024-05-08T16:22:34.719 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj223 td/test-erasure-eio/CHECK 2024-05-08T16:22:34.747 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:22:34.748 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:22:34.748 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj224 = obj250 ']' 2024-05-08T16:22:34.748 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj224 td/test-erasure-eio/CHECK 2024-05-08T16:22:34.776 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:22:34.777 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:22:34.777 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj225 = obj250 ']' 2024-05-08T16:22:34.777 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj225 td/test-erasure-eio/CHECK 2024-05-08T16:22:34.805 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:22:34.806 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:22:34.806 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj226 = obj250 ']' 2024-05-08T16:22:34.807 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj226 td/test-erasure-eio/CHECK 2024-05-08T16:22:34.838 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:22:34.840 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:22:34.840 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj227 = obj250 ']' 2024-05-08T16:22:34.840 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj227 td/test-erasure-eio/CHECK 2024-05-08T16:22:34.865 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:22:34.867 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:22:34.867 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj228 = obj250 ']' 2024-05-08T16:22:34.867 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj228 td/test-erasure-eio/CHECK 2024-05-08T16:22:34.897 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:22:34.898 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:22:34.898 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj229 = obj250 ']' 2024-05-08T16:22:34.898 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj229 td/test-erasure-eio/CHECK 2024-05-08T16:22:34.923 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:22:34.924 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:22:34.924 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj230 = obj250 ']' 2024-05-08T16:22:34.925 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj230 td/test-erasure-eio/CHECK 2024-05-08T16:22:34.953 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:22:34.954 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:22:34.954 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj231 = obj250 ']' 2024-05-08T16:22:34.954 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj231 td/test-erasure-eio/CHECK 2024-05-08T16:22:34.985 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:22:34.985 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:22:34.985 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj232 = obj250 ']' 2024-05-08T16:22:34.986 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj232 td/test-erasure-eio/CHECK 2024-05-08T16:22:35.016 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:22:35.017 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:22:35.017 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj233 = obj250 ']' 2024-05-08T16:22:35.017 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj233 td/test-erasure-eio/CHECK 2024-05-08T16:22:35.039 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:22:35.040 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:22:35.040 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj234 = obj250 ']' 2024-05-08T16:22:35.040 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj234 td/test-erasure-eio/CHECK 2024-05-08T16:22:35.075 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:22:35.076 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:22:35.076 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj235 = obj250 ']' 2024-05-08T16:22:35.077 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj235 td/test-erasure-eio/CHECK 2024-05-08T16:22:35.099 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:22:35.100 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:22:35.100 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj236 = obj250 ']' 2024-05-08T16:22:35.100 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj236 td/test-erasure-eio/CHECK 2024-05-08T16:22:35.127 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:22:35.128 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:22:35.128 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj237 = obj250 ']' 2024-05-08T16:22:35.128 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj237 td/test-erasure-eio/CHECK 2024-05-08T16:22:35.156 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:22:35.157 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:22:35.157 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj238 = obj250 ']' 2024-05-08T16:22:35.157 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj238 td/test-erasure-eio/CHECK 2024-05-08T16:22:35.187 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:22:35.188 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:22:35.188 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj239 = obj250 ']' 2024-05-08T16:22:35.188 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj239 td/test-erasure-eio/CHECK 2024-05-08T16:22:35.215 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:22:35.216 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:22:35.216 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj240 = obj250 ']' 2024-05-08T16:22:35.216 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj240 td/test-erasure-eio/CHECK 2024-05-08T16:22:35.245 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:22:35.245 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:22:35.245 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj241 = obj250 ']' 2024-05-08T16:22:35.245 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj241 td/test-erasure-eio/CHECK 2024-05-08T16:22:35.274 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:22:35.275 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:22:35.275 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj242 = obj250 ']' 2024-05-08T16:22:35.275 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj242 td/test-erasure-eio/CHECK 2024-05-08T16:22:35.303 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:22:35.304 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:22:35.304 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj243 = obj250 ']' 2024-05-08T16:22:35.304 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj243 td/test-erasure-eio/CHECK 2024-05-08T16:22:35.332 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:22:35.333 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:22:35.333 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj244 = obj250 ']' 2024-05-08T16:22:35.333 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj244 td/test-erasure-eio/CHECK 2024-05-08T16:22:35.361 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:22:35.363 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:22:35.363 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj245 = obj250 ']' 2024-05-08T16:22:35.363 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj245 td/test-erasure-eio/CHECK 2024-05-08T16:22:35.390 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:22:35.391 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:22:35.392 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj246 = obj250 ']' 2024-05-08T16:22:35.392 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj246 td/test-erasure-eio/CHECK 2024-05-08T16:22:35.419 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:22:35.420 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:22:35.420 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj247 = obj250 ']' 2024-05-08T16:22:35.421 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj247 td/test-erasure-eio/CHECK 2024-05-08T16:22:35.449 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:22:35.449 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:22:35.450 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj248 = obj250 ']' 2024-05-08T16:22:35.450 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj248 td/test-erasure-eio/CHECK 2024-05-08T16:22:35.478 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:22:35.479 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:22:35.479 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj249 = obj250 ']' 2024-05-08T16:22:35.479 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj249 td/test-erasure-eio/CHECK 2024-05-08T16:22:35.506 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:22:35.507 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:22:35.507 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj250 = obj250 ']' 2024-05-08T16:22:35.507 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:595: TEST_ec_backfill_unfound: rados -p pool-jerasure get obj250 td/test-erasure-eio/CHECK 2024-05-08T16:22:35.533 INFO:tasks.workunit.client.0.smithi172.stderr:error getting pool-jerasure/obj250: (2) No such file or directory 2024-05-08T16:22:35.535 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:22:35.535 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj251 = obj250 ']' 2024-05-08T16:22:35.535 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj251 td/test-erasure-eio/CHECK 2024-05-08T16:22:35.563 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:22:35.564 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:22:35.564 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj252 = obj250 ']' 2024-05-08T16:22:35.564 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj252 td/test-erasure-eio/CHECK 2024-05-08T16:22:35.592 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:22:35.593 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:22:35.593 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj253 = obj250 ']' 2024-05-08T16:22:35.593 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj253 td/test-erasure-eio/CHECK 2024-05-08T16:22:35.621 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:22:35.622 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:22:35.622 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj254 = obj250 ']' 2024-05-08T16:22:35.622 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj254 td/test-erasure-eio/CHECK 2024-05-08T16:22:35.652 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:22:35.653 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:22:35.653 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj255 = obj250 ']' 2024-05-08T16:22:35.654 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj255 td/test-erasure-eio/CHECK 2024-05-08T16:22:35.680 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:22:35.681 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:22:35.681 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj256 = obj250 ']' 2024-05-08T16:22:35.681 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj256 td/test-erasure-eio/CHECK 2024-05-08T16:22:35.734 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:22:35.735 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:22:35.735 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj257 = obj250 ']' 2024-05-08T16:22:35.735 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj257 td/test-erasure-eio/CHECK 2024-05-08T16:22:35.763 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:22:35.764 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:22:35.764 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj258 = obj250 ']' 2024-05-08T16:22:35.764 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj258 td/test-erasure-eio/CHECK 2024-05-08T16:22:35.792 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:22:35.793 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:22:35.793 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj259 = obj250 ']' 2024-05-08T16:22:35.793 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj259 td/test-erasure-eio/CHECK 2024-05-08T16:22:35.822 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:22:35.822 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:22:35.822 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj260 = obj250 ']' 2024-05-08T16:22:35.823 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj260 td/test-erasure-eio/CHECK 2024-05-08T16:22:35.851 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:22:35.852 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:22:35.852 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj261 = obj250 ']' 2024-05-08T16:22:35.852 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj261 td/test-erasure-eio/CHECK 2024-05-08T16:22:35.881 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:22:35.881 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:22:35.881 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj262 = obj250 ']' 2024-05-08T16:22:35.881 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj262 td/test-erasure-eio/CHECK 2024-05-08T16:22:35.909 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:22:35.910 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:22:35.910 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj263 = obj250 ']' 2024-05-08T16:22:35.910 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj263 td/test-erasure-eio/CHECK 2024-05-08T16:22:35.938 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:22:35.939 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:22:35.939 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj264 = obj250 ']' 2024-05-08T16:22:35.939 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj264 td/test-erasure-eio/CHECK 2024-05-08T16:22:35.968 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:22:35.968 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:22:35.968 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj265 = obj250 ']' 2024-05-08T16:22:35.968 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj265 td/test-erasure-eio/CHECK 2024-05-08T16:22:35.997 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:22:35.998 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:22:35.998 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj266 = obj250 ']' 2024-05-08T16:22:35.998 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj266 td/test-erasure-eio/CHECK 2024-05-08T16:22:36.027 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:22:36.027 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:22:36.027 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj267 = obj250 ']' 2024-05-08T16:22:36.027 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj267 td/test-erasure-eio/CHECK 2024-05-08T16:22:36.055 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:22:36.056 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:22:36.056 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj268 = obj250 ']' 2024-05-08T16:22:36.056 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj268 td/test-erasure-eio/CHECK 2024-05-08T16:22:36.084 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:22:36.085 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:22:36.085 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj269 = obj250 ']' 2024-05-08T16:22:36.085 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj269 td/test-erasure-eio/CHECK 2024-05-08T16:22:36.113 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:22:36.114 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:22:36.114 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj270 = obj250 ']' 2024-05-08T16:22:36.114 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj270 td/test-erasure-eio/CHECK 2024-05-08T16:22:36.175 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:22:36.176 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:22:36.176 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj271 = obj250 ']' 2024-05-08T16:22:36.176 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj271 td/test-erasure-eio/CHECK 2024-05-08T16:22:36.204 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:22:36.205 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:22:36.206 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj272 = obj250 ']' 2024-05-08T16:22:36.206 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj272 td/test-erasure-eio/CHECK 2024-05-08T16:22:36.261 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:22:36.262 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:22:36.262 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj273 = obj250 ']' 2024-05-08T16:22:36.262 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj273 td/test-erasure-eio/CHECK 2024-05-08T16:22:36.290 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:22:36.291 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:22:36.291 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj274 = obj250 ']' 2024-05-08T16:22:36.291 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj274 td/test-erasure-eio/CHECK 2024-05-08T16:22:36.320 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:22:36.320 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:22:36.320 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj275 = obj250 ']' 2024-05-08T16:22:36.321 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj275 td/test-erasure-eio/CHECK 2024-05-08T16:22:36.349 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:22:36.350 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:22:36.350 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj276 = obj250 ']' 2024-05-08T16:22:36.350 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj276 td/test-erasure-eio/CHECK 2024-05-08T16:22:36.379 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:22:36.380 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:22:36.380 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj277 = obj250 ']' 2024-05-08T16:22:36.380 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj277 td/test-erasure-eio/CHECK 2024-05-08T16:22:36.409 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:22:36.409 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:22:36.410 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj278 = obj250 ']' 2024-05-08T16:22:36.410 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj278 td/test-erasure-eio/CHECK 2024-05-08T16:22:36.438 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:22:36.439 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:22:36.439 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj279 = obj250 ']' 2024-05-08T16:22:36.439 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj279 td/test-erasure-eio/CHECK 2024-05-08T16:22:36.467 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:22:36.468 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:22:36.468 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj280 = obj250 ']' 2024-05-08T16:22:36.468 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj280 td/test-erasure-eio/CHECK 2024-05-08T16:22:36.496 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:22:36.497 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:22:36.497 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj281 = obj250 ']' 2024-05-08T16:22:36.498 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj281 td/test-erasure-eio/CHECK 2024-05-08T16:22:36.526 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:22:36.526 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:22:36.527 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj282 = obj250 ']' 2024-05-08T16:22:36.527 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj282 td/test-erasure-eio/CHECK 2024-05-08T16:22:36.555 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:22:36.556 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:22:36.556 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj283 = obj250 ']' 2024-05-08T16:22:36.556 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj283 td/test-erasure-eio/CHECK 2024-05-08T16:22:36.584 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:22:36.585 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:22:36.585 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj284 = obj250 ']' 2024-05-08T16:22:36.585 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj284 td/test-erasure-eio/CHECK 2024-05-08T16:22:36.613 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:22:36.614 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:22:36.614 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj285 = obj250 ']' 2024-05-08T16:22:36.614 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj285 td/test-erasure-eio/CHECK 2024-05-08T16:22:36.643 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:22:36.644 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:22:36.644 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj286 = obj250 ']' 2024-05-08T16:22:36.644 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj286 td/test-erasure-eio/CHECK 2024-05-08T16:22:36.672 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:22:36.672 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:22:36.673 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj287 = obj250 ']' 2024-05-08T16:22:36.673 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj287 td/test-erasure-eio/CHECK 2024-05-08T16:22:36.701 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:22:36.701 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:22:36.702 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj288 = obj250 ']' 2024-05-08T16:22:36.702 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj288 td/test-erasure-eio/CHECK 2024-05-08T16:22:36.729 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:22:36.730 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:22:36.730 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj289 = obj250 ']' 2024-05-08T16:22:36.730 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj289 td/test-erasure-eio/CHECK 2024-05-08T16:22:36.758 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:22:36.759 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:22:36.759 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj290 = obj250 ']' 2024-05-08T16:22:36.760 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj290 td/test-erasure-eio/CHECK 2024-05-08T16:22:36.787 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:22:36.788 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:22:36.788 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj291 = obj250 ']' 2024-05-08T16:22:36.788 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj291 td/test-erasure-eio/CHECK 2024-05-08T16:22:36.818 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:22:36.818 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:22:36.818 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj292 = obj250 ']' 2024-05-08T16:22:36.819 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj292 td/test-erasure-eio/CHECK 2024-05-08T16:22:36.847 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:22:36.847 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:22:36.847 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj293 = obj250 ']' 2024-05-08T16:22:36.848 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj293 td/test-erasure-eio/CHECK 2024-05-08T16:22:36.876 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:22:36.876 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:22:36.877 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj294 = obj250 ']' 2024-05-08T16:22:36.877 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj294 td/test-erasure-eio/CHECK 2024-05-08T16:22:36.905 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:22:36.906 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:22:36.906 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj295 = obj250 ']' 2024-05-08T16:22:36.906 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj295 td/test-erasure-eio/CHECK 2024-05-08T16:22:36.934 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:22:36.935 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:22:36.935 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj296 = obj250 ']' 2024-05-08T16:22:36.935 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj296 td/test-erasure-eio/CHECK 2024-05-08T16:22:36.963 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:22:36.964 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:22:36.964 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj297 = obj250 ']' 2024-05-08T16:22:36.964 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj297 td/test-erasure-eio/CHECK 2024-05-08T16:22:36.992 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:22:36.993 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:22:36.993 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj298 = obj250 ']' 2024-05-08T16:22:36.993 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj298 td/test-erasure-eio/CHECK 2024-05-08T16:22:37.021 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:22:37.022 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:22:37.022 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj299 = obj250 ']' 2024-05-08T16:22:37.022 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj299 td/test-erasure-eio/CHECK 2024-05-08T16:22:37.050 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:22:37.051 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:591: TEST_ec_backfill_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:22:37.051 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:593: TEST_ec_backfill_unfound: '[' obj300 = obj250 ']' 2024-05-08T16:22:37.051 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:597: TEST_ec_backfill_unfound: rados --pool pool-jerasure get obj300 td/test-erasure-eio/CHECK 2024-05-08T16:22:37.080 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:598: TEST_ec_backfill_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:22:37.080 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:602: TEST_ec_backfill_unfound: rm -f td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:22:37.081 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:604: TEST_ec_backfill_unfound: delete_erasure_coded_pool pool-jerasure 2024-05-08T16:22:37.082 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:84: delete_erasure_coded_pool: local poolname=pool-jerasure 2024-05-08T16:22:37.082 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:85: delete_erasure_coded_pool: ceph osd pool delete pool-jerasure pool-jerasure --yes-i-really-really-mean-it 2024-05-08T16:22:37.595 INFO:tasks.workunit.client.0.smithi172.stderr:pool 'pool-jerasure' does not exist 2024-05-08T16:22:37.611 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:86: delete_erasure_coded_pool: ceph osd erasure-code-profile rm myprofile 2024-05-08T16:22:38.491 INFO:tasks.workunit.client.0.smithi172.stderr:erasure-code-profile myprofile does not exist 2024-05-08T16:22:38.510 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:42: run: teardown td/test-erasure-eio 2024-05-08T16:22:38.511 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-eio 2024-05-08T16:22:38.511 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-05-08T16:22:38.511 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-eio KILL 2024-05-08T16:22:38.511 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-08T16:22:38.512 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-08T16:22:38.512 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-08T16:22:38.512 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-08T16:22:38.512 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-08T16:22:38.823 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-08T16:22:38.823 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-05-08T16:22:38.824 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-05-08T16:22:38.825 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-05-08T16:22:38.827 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-05-08T16:22:38.827 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-05-08T16:22:38.828 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-05-08T16:22:38.829 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-08T16:22:38.829 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-05-08T16:22:38.829 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-05-08T16:22:38.829 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-08T16:22:38.830 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-05-08T16:22:38.832 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-05-08T16:22:38.832 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-eio 2024-05-08T16:22:38.902 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-05-08T16:22:38.902 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T16:22:38.902 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81110 2024-05-08T16:22:38.902 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.81110 2024-05-08T16:22:38.904 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-05-08T16:22:38.904 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-05-08T16:22:38.904 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:32: run: for func in $funcs 2024-05-08T16:22:38.904 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:33: run: setup td/test-erasure-eio 2024-05-08T16:22:38.904 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/test-erasure-eio 2024-05-08T16:22:38.904 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/test-erasure-eio 2024-05-08T16:22:38.904 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-eio 2024-05-08T16:22:38.905 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-05-08T16:22:38.905 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-eio KILL 2024-05-08T16:22:38.905 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-08T16:22:38.905 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-08T16:22:38.905 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-08T16:22:38.905 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-08T16:22:38.905 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-08T16:22:38.906 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-08T16:22:38.907 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-05-08T16:22:38.908 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-05-08T16:22:38.908 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-05-08T16:22:38.909 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-05-08T16:22:38.910 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-05-08T16:22:38.910 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-05-08T16:22:38.911 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-08T16:22:38.911 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-05-08T16:22:38.911 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-05-08T16:22:38.912 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-08T16:22:38.912 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-05-08T16:22:38.914 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-05-08T16:22:38.914 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-eio 2024-05-08T16:22:38.915 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-05-08T16:22:38.915 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T16:22:38.915 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81110 2024-05-08T16:22:38.915 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.81110 2024-05-08T16:22:38.916 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-05-08T16:22:38.916 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-05-08T16:22:38.916 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/test-erasure-eio 2024-05-08T16:22:38.918 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-05-08T16:22:38.918 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T16:22:38.918 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81110 2024-05-08T16:22:38.918 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.81110 2024-05-08T16:22:38.920 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-05-08T16:22:38.921 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-05-08T16:22:38.921 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-05-08T16:22:38.921 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/test-erasure-eio 1' TERM HUP INT 2024-05-08T16:22:38.921 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:34: run: run_mon td/test-erasure-eio a 2024-05-08T16:22:38.921 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/test-erasure-eio 2024-05-08T16:22:38.921 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-05-08T16:22:38.921 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-05-08T16:22:38.921 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-05-08T16:22:38.921 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/test-erasure-eio/a 2024-05-08T16:22:38.921 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-eio/a --run-dir=td/test-erasure-eio 2024-05-08T16:22:39.351 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-05-08T16:22:39.352 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-08T16:22:39.352 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-08T16:22:39.352 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-08T16:22:39.352 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T16:22:39.352 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81110 2024-05-08T16:22:39.353 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.81110/$cluster-$name.asok' 2024-05-08T16:22:39.353 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-eio/a '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.81110/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-eio/log --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-05-08T16:22:39.580 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-05-08T16:22:39.581 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-05-08T16:22:39.581 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-05-08T16:22:39.581 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-05-08T16:22:39.581 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-05-08T16:22:39.582 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-05-08T16:22:39.583 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-05-08T16:22:39.583 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-05-08T16:22:39.583 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-05-08T16:22:39.583 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-08T16:22:39.583 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T16:22:39.583 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81110 2024-05-08T16:22:39.583 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.81110/ceph-mon.a.asok 2024-05-08T16:22:39.583 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-05-08T16:22:39.584 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.81110/ceph-mon.a.asok config get fsid 2024-05-08T16:22:39.698 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-05-08T16:22:39.699 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-05-08T16:22:39.699 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-05-08T16:22:39.699 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-05-08T16:22:39.700 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-05-08T16:22:39.700 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-05-08T16:22:39.700 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-05-08T16:22:39.700 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-05-08T16:22:39.700 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-08T16:22:39.700 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T16:22:39.700 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81110 2024-05-08T16:22:39.700 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.81110/ceph-mon.a.asok 2024-05-08T16:22:39.700 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-05-08T16:22:39.700 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.81110/ceph-mon.a.asok config get mon_host 2024-05-08T16:22:39.805 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:35: run: run_mgr td/test-erasure-eio x 2024-05-08T16:22:39.805 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/test-erasure-eio 2024-05-08T16:22:39.805 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-05-08T16:22:39.805 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-05-08T16:22:39.806 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-05-08T16:22:39.806 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/test-erasure-eio/x 2024-05-08T16:22:39.806 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-05-08T16:22:40.717 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-05-08T16:22:40.717 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-08T16:22:40.717 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-08T16:22:40.718 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-08T16:22:40.718 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T16:22:40.718 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81110 2024-05-08T16:22:40.718 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.81110/$cluster-$name.asok' 2024-05-08T16:22:40.719 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-05-08T16:22:40.720 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-eio/x '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.81110/$cluster-$name.asok' --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-05-08T16:22:40.753 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:36: run: create_pool rbd 4 2024-05-08T16:22:40.754 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-05-08T16:22:41.107 INFO:tasks.workunit.client.0.smithi172.stderr:pool 'rbd' already exists 2024-05-08T16:22:41.126 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-05-08T16:22:42.128 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: get_asok_path mon.a 2024-05-08T16:22:42.128 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-05-08T16:22:42.128 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-05-08T16:22:42.128 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-08T16:22:42.129 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T16:22:42.129 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81110 2024-05-08T16:22:42.129 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.81110/ceph-mon.a.asok 2024-05-08T16:22:42.129 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: CEPH_ARGS= 2024-05-08T16:22:42.129 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: ceph --admin-daemon /tmp/ceph-asok.81110/ceph-mon.a.asok log flush 2024-05-08T16:22:42.238 INFO:tasks.workunit.client.0.smithi172.stdout:{} 2024-05-08T16:22:42.251 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:40: run: grep 'load: jerasure.*lrc' td/test-erasure-eio/mon.a.log 2024-05-08T16:22:42.253 INFO:tasks.workunit.client.0.smithi172.stdout:2024-05-08T16:22:39.399+0000 7fcbc8199c80 0 load: jerasure load: lrc 2024-05-08T16:22:42.254 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:41: run: TEST_ec_object_attr_read_error td/test-erasure-eio 2024-05-08T16:22:42.254 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:359: TEST_ec_object_attr_read_error: local dir=td/test-erasure-eio 2024-05-08T16:22:42.254 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:360: TEST_ec_object_attr_read_error: local objname=myobject 2024-05-08T16:22:42.254 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:362: TEST_ec_object_attr_read_error: setup_osds 7 2024-05-08T16:22:42.254 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:47: setup_osds: local count=7 2024-05-08T16:22:42.254 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:48: setup_osds: shift 2024-05-08T16:22:42.255 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: expr 7 - 1 2024-05-08T16:22:42.256 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: seq 0 6 2024-05-08T16:22:42.261 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-05-08T16:22:42.261 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 0 2024-05-08T16:22:42.261 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-05-08T16:22:42.261 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-05-08T16:22:42.261 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-05-08T16:22:42.261 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-05-08T16:22:42.261 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/0 2024-05-08T16:22:42.261 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=4457b0f1-028b-482b-8ee3-7ee05861e6cc --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-05-08T16:22:42.261 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-08T16:22:42.261 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-05-08T16:22:42.261 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-08T16:22:42.261 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/0' 2024-05-08T16:22:42.262 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/0/journal' 2024-05-08T16:22:42.262 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-05-08T16:22:42.262 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-05-08T16:22:42.262 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-05-08T16:22:42.262 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-05-08T16:22:42.262 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-08T16:22:42.262 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-08T16:22:42.262 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-08T16:22:42.262 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T16:22:42.262 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81110 2024-05-08T16:22:42.262 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.81110/$cluster-$name.asok' 2024-05-08T16:22:42.262 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.81110/$cluster-$name.asok' 2024-05-08T16:22:42.262 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-05-08T16:22:42.263 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-05-08T16:22:42.263 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-05-08T16:22:42.263 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-05-08T16:22:42.263 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-05-08T16:22:42.263 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-08T16:22:42.263 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-08T16:22:42.263 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-08T16:22:42.263 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-08T16:22:42.263 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-05-08T16:22:42.263 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-05-08T16:22:42.263 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/0 2024-05-08T16:22:42.263 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-05-08T16:22:42.263 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=205cf4f7-4f08-4a92-9377-a985a76cb275 2024-05-08T16:22:42.264 INFO:tasks.workunit.client.0.smithi172.stdout:add osd0 205cf4f7-4f08-4a92-9377-a985a76cb275 2024-05-08T16:22:42.264 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 205cf4f7-4f08-4a92-9377-a985a76cb275' 2024-05-08T16:22:42.264 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-05-08T16:22:42.286 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDSpjtmT3xxERAAi6K+410j8w/K0pSzZkwykQ== 2024-05-08T16:22:42.286 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDSpjtmT3xxERAAi6K+410j8w/K0pSzZkwykQ=="}' 2024-05-08T16:22:42.286 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 205cf4f7-4f08-4a92-9377-a985a76cb275 -i td/test-erasure-eio/0/new.json 2024-05-08T16:22:42.642 INFO:tasks.workunit.client.0.smithi172.stdout:0 2024-05-08T16:22:42.658 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/0/new.json 2024-05-08T16:22:42.659 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=4457b0f1-028b-482b-8ee3-7ee05861e6cc --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81110/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDSpjtmT3xxERAAi6K+410j8w/K0pSzZkwykQ== --osd-uuid 205cf4f7-4f08-4a92-9377-a985a76cb275 2024-05-08T16:22:42.690 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:22:42.695+0000 7f6367eae7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:22:42.718 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:22:42.723+0000 7f6367eae7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:22:42.719 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:22:42.727+0000 7f6367eae7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:22:42.720 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:22:42.727+0000 7f6367eae7c0 -1 bluestore(td/test-erasure-eio/0/block) _read_bdev_label failed to open td/test-erasure-eio/0/block: (2) No such file or directory 2024-05-08T16:22:42.720 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:22:42.727+0000 7f6367eae7c0 -1 bluestore(td/test-erasure-eio/0/block) _read_bdev_label failed to open td/test-erasure-eio/0/block: (2) No such file or directory 2024-05-08T16:22:42.720 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:22:42.727+0000 7f6367eae7c0 -1 bluestore(td/test-erasure-eio/0/block) _read_bdev_label failed to open td/test-erasure-eio/0/block: (2) No such file or directory 2024-05-08T16:22:42.758 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:22:42.763+0000 7f6367eae7c0 -1 bluestore(td/test-erasure-eio/0) _read_fsid unparsable uuid 2024-05-08T16:22:50.385 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/0/keyring 2024-05-08T16:22:50.385 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-05-08T16:22:50.386 INFO:tasks.workunit.client.0.smithi172.stdout:adding osd0 key to auth repository 2024-05-08T16:22:50.387 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-05-08T16:22:50.387 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-05-08T16:22:50.950 INFO:tasks.workunit.client.0.smithi172.stdout:start osd.0 2024-05-08T16:22:50.950 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-05-08T16:22:50.951 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=4457b0f1-028b-482b-8ee3-7ee05861e6cc --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81110/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-05-08T16:22:50.951 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-05-08T16:22:50.951 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-05-08T16:22:50.951 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-05-08T16:22:50.981 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:22:50.986+0000 7f8507dc67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:22:50.983 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:22:50.986+0000 7f8507dc67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:22:50.984 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:22:50.990+0000 7f8507dc67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:22:51.391 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-05-08T16:22:51.391 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-08T16:22:51.391 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-05-08T16:22:51.391 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-08T16:22:51.391 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-08T16:22:51.391 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:22:51.391 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-08T16:22:51.391 INFO:tasks.workunit.client.0.smithi172.stdout:0 2024-05-08T16:22:51.392 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:22:51.392 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:22:51.553 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:22:51.558+0000 7f8507dc67c0 -1 Falling back to public interface 2024-05-08T16:22:51.845 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:22:52.846 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:22:52.847 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:22:52.847 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-08T16:22:52.847 INFO:tasks.workunit.client.0.smithi172.stdout:1 2024-05-08T16:22:52.847 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:22:52.848 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:22:53.258 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:22:54.259 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:22:54.259 INFO:tasks.workunit.client.0.smithi172.stdout:2 2024-05-08T16:22:54.260 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:22:54.260 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-08T16:22:54.260 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:22:54.260 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:22:54.263 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:22:54.266+0000 7f8507dc67c0 -1 osd.0 0 log_to_monitors true 2024-05-08T16:22:54.800 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:22:55.803 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:22:55.803 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:22:55.803 INFO:tasks.workunit.client.0.smithi172.stdout:3 2024-05-08T16:22:55.804 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-05-08T16:22:55.806 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:22:55.806 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:22:56.254 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:22:57.256 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:22:57.256 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:22:57.256 INFO:tasks.workunit.client.0.smithi172.stdout:4 2024-05-08T16:22:57.257 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-05-08T16:22:57.257 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:22:57.257 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:22:57.687 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:22:58.688 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:22:58.688 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:22:58.689 INFO:tasks.workunit.client.0.smithi172.stdout:5 2024-05-08T16:22:58.689 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-05-08T16:22:58.689 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:22:58.689 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:22:59.191 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:23:00.193 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:23:00.193 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:23:00.193 INFO:tasks.workunit.client.0.smithi172.stdout:6 2024-05-08T16:23:00.193 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-05-08T16:23:00.194 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:23:00.194 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:23:00.676 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:23:01.678 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:23:01.678 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:23:01.678 INFO:tasks.workunit.client.0.smithi172.stdout:7 2024-05-08T16:23:01.679 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-05-08T16:23:01.679 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:23:01.679 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:23:02.210 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:23:03.211 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:23:03.211 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:23:03.211 INFO:tasks.workunit.client.0.smithi172.stdout:8 2024-05-08T16:23:03.212 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-05-08T16:23:03.212 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:23:03.212 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:23:03.685 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:23:04.687 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:23:04.687 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:23:04.688 INFO:tasks.workunit.client.0.smithi172.stdout:9 2024-05-08T16:23:04.688 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-05-08T16:23:04.688 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:23:04.688 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:23:05.251 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:23:06.252 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:23:06.252 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:23:06.252 INFO:tasks.workunit.client.0.smithi172.stdout:10 2024-05-08T16:23:06.253 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-05-08T16:23:06.253 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:23:06.253 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:23:06.696 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:23:07.697 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:23:07.698 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:23:07.698 INFO:tasks.workunit.client.0.smithi172.stdout:11 2024-05-08T16:23:07.698 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-05-08T16:23:07.699 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:23:07.699 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:23:08.297 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:23:09.299 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:23:09.299 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:23:09.300 INFO:tasks.workunit.client.0.smithi172.stdout:12 2024-05-08T16:23:09.300 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-05-08T16:23:09.300 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:23:09.300 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:23:09.848 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:23:10.849 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:23:10.850 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:23:10.850 INFO:tasks.workunit.client.0.smithi172.stdout:13 2024-05-08T16:23:10.850 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-05-08T16:23:10.850 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:23:10.851 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:23:11.635 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:23:12.637 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:23:12.637 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:23:12.637 INFO:tasks.workunit.client.0.smithi172.stdout:14 2024-05-08T16:23:12.638 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-05-08T16:23:12.638 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:23:12.638 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:23:13.793 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:23:14.795 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:23:14.795 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:23:14.796 INFO:tasks.workunit.client.0.smithi172.stdout:15 2024-05-08T16:23:14.796 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-05-08T16:23:14.796 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:23:14.797 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:23:15.808 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:23:16.810 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:23:16.810 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:23:16.811 INFO:tasks.workunit.client.0.smithi172.stdout:16 2024-05-08T16:23:16.811 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-05-08T16:23:16.811 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:23:16.811 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:23:17.832 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:23:18.833 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:23:18.834 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:23:18.834 INFO:tasks.workunit.client.0.smithi172.stdout:17 2024-05-08T16:23:18.835 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-05-08T16:23:18.835 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:23:18.835 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:23:19.882 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:23:20.884 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:23:20.884 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:23:20.884 INFO:tasks.workunit.client.0.smithi172.stdout:18 2024-05-08T16:23:20.885 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-05-08T16:23:20.885 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:23:20.885 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:23:21.601 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:23:22.602 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:23:22.602 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:23:22.602 INFO:tasks.workunit.client.0.smithi172.stdout:19 2024-05-08T16:23:22.603 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-05-08T16:23:22.603 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:23:22.603 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:23:23.106 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:23:24.108 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:23:24.108 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:23:24.109 INFO:tasks.workunit.client.0.smithi172.stdout:20 2024-05-08T16:23:24.109 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-05-08T16:23:24.109 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:23:24.109 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:23:24.704 INFO:tasks.workunit.client.0.smithi172.stdout:osd.0 up in weight 1 up_from 6 up_thru 6 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/4189797646,v1:127.0.0.1:6803/4189797646] [v2:127.0.0.1:6804/4189797646,v1:127.0.0.1:6805/4189797646] exists,up 205cf4f7-4f08-4a92-9377-a985a76cb275 2024-05-08T16:23:24.708 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-08T16:23:24.708 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-08T16:23:24.709 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-08T16:23:24.709 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-05-08T16:23:24.709 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 1 2024-05-08T16:23:24.709 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-05-08T16:23:24.709 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-05-08T16:23:24.709 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-05-08T16:23:24.709 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-05-08T16:23:24.709 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/1 2024-05-08T16:23:24.709 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=4457b0f1-028b-482b-8ee3-7ee05861e6cc --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-05-08T16:23:24.709 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-08T16:23:24.710 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-05-08T16:23:24.710 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-08T16:23:24.710 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/1' 2024-05-08T16:23:24.710 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/1/journal' 2024-05-08T16:23:24.710 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-05-08T16:23:24.710 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-05-08T16:23:24.710 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-05-08T16:23:24.710 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-05-08T16:23:24.710 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-08T16:23:24.710 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-08T16:23:24.710 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-08T16:23:24.710 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T16:23:24.710 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81110 2024-05-08T16:23:24.710 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.81110/$cluster-$name.asok' 2024-05-08T16:23:24.711 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.81110/$cluster-$name.asok' 2024-05-08T16:23:24.711 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-05-08T16:23:24.711 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-05-08T16:23:24.711 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-05-08T16:23:24.711 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-05-08T16:23:24.711 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-05-08T16:23:24.711 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-08T16:23:24.711 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-08T16:23:24.711 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-08T16:23:24.711 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-08T16:23:24.711 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-05-08T16:23:24.711 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-05-08T16:23:24.711 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/1 2024-05-08T16:23:24.712 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-05-08T16:23:24.712 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=44cea6e4-14d4-45e8-8eed-6bbcdcfa4563 2024-05-08T16:23:24.712 INFO:tasks.workunit.client.0.smithi172.stdout:add osd1 44cea6e4-14d4-45e8-8eed-6bbcdcfa4563 2024-05-08T16:23:24.713 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 44cea6e4-14d4-45e8-8eed-6bbcdcfa4563' 2024-05-08T16:23:24.713 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-05-08T16:23:24.735 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQD8pjtmPn8ALBAAEEA5tq90nxlA9AzdD7IaZQ== 2024-05-08T16:23:24.735 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQD8pjtmPn8ALBAAEEA5tq90nxlA9AzdD7IaZQ=="}' 2024-05-08T16:23:24.735 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 44cea6e4-14d4-45e8-8eed-6bbcdcfa4563 -i td/test-erasure-eio/1/new.json 2024-05-08T16:23:25.315 INFO:tasks.workunit.client.0.smithi172.stdout:1 2024-05-08T16:23:25.331 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/1/new.json 2024-05-08T16:23:25.332 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=4457b0f1-028b-482b-8ee3-7ee05861e6cc --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81110/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQD8pjtmPn8ALBAAEEA5tq90nxlA9AzdD7IaZQ== --osd-uuid 44cea6e4-14d4-45e8-8eed-6bbcdcfa4563 2024-05-08T16:23:25.360 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:23:25.365+0000 7fa025a557c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:23:25.362 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:23:25.365+0000 7fa025a557c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:23:25.363 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:23:25.365+0000 7fa025a557c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:23:25.364 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:23:25.369+0000 7fa025a557c0 -1 bluestore(td/test-erasure-eio/1/block) _read_bdev_label failed to open td/test-erasure-eio/1/block: (2) No such file or directory 2024-05-08T16:23:25.364 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:23:25.369+0000 7fa025a557c0 -1 bluestore(td/test-erasure-eio/1/block) _read_bdev_label failed to open td/test-erasure-eio/1/block: (2) No such file or directory 2024-05-08T16:23:25.364 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:23:25.369+0000 7fa025a557c0 -1 bluestore(td/test-erasure-eio/1/block) _read_bdev_label failed to open td/test-erasure-eio/1/block: (2) No such file or directory 2024-05-08T16:23:25.439 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:23:25.441+0000 7fa025a557c0 -1 bluestore(td/test-erasure-eio/1) _read_fsid unparsable uuid 2024-05-08T16:23:32.838 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/1/keyring 2024-05-08T16:23:32.838 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-05-08T16:23:32.839 INFO:tasks.workunit.client.0.smithi172.stdout:adding osd1 key to auth repository 2024-05-08T16:23:32.839 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-05-08T16:23:32.839 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-05-08T16:23:33.389 INFO:tasks.workunit.client.0.smithi172.stdout:start osd.1 2024-05-08T16:23:33.390 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-05-08T16:23:33.390 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-05-08T16:23:33.390 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-05-08T16:23:33.390 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=4457b0f1-028b-482b-8ee3-7ee05861e6cc --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81110/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-05-08T16:23:33.390 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-05-08T16:23:33.422 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:23:33.425+0000 7fd8e0b647c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:23:33.425 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:23:33.429+0000 7fd8e0b647c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:23:33.426 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:23:33.429+0000 7fd8e0b647c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:23:33.811 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-05-08T16:23:33.811 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-08T16:23:33.812 INFO:tasks.workunit.client.0.smithi172.stdout:0 2024-05-08T16:23:33.812 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-05-08T16:23:33.812 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-08T16:23:33.812 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-08T16:23:33.813 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:23:33.813 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-08T16:23:33.813 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:23:33.813 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T16:23:33.977 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:23:33.981+0000 7fd8e0b647c0 -1 Falling back to public interface 2024-05-08T16:23:34.258 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:23:35.260 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:23:35.260 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:23:35.260 INFO:tasks.workunit.client.0.smithi172.stdout:1 2024-05-08T16:23:35.261 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-08T16:23:35.261 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:23:35.262 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T16:23:35.682 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:23:36.683 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:23:36.683 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:23:36.684 INFO:tasks.workunit.client.0.smithi172.stdout:2 2024-05-08T16:23:36.684 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-08T16:23:36.684 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:23:36.684 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T16:23:36.770 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:23:36.773+0000 7fd8e0b647c0 -1 osd.1 0 log_to_monitors true 2024-05-08T16:23:37.189 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:23:38.190 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:23:38.190 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:23:38.191 INFO:tasks.workunit.client.0.smithi172.stdout:3 2024-05-08T16:23:38.191 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-05-08T16:23:38.191 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:23:38.191 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T16:23:38.670 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:23:39.671 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:23:39.671 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:23:39.671 INFO:tasks.workunit.client.0.smithi172.stdout:4 2024-05-08T16:23:39.672 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-05-08T16:23:39.672 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:23:39.672 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T16:23:40.368 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:23:41.369 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:23:41.369 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:23:41.370 INFO:tasks.workunit.client.0.smithi172.stdout:5 2024-05-08T16:23:41.370 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-05-08T16:23:41.370 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:23:41.370 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T16:23:42.018 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:23:43.020 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:23:43.020 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:23:43.020 INFO:tasks.workunit.client.0.smithi172.stdout:6 2024-05-08T16:23:43.021 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-05-08T16:23:43.021 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:23:43.021 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T16:23:43.459 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:23:44.461 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:23:44.461 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:23:44.461 INFO:tasks.workunit.client.0.smithi172.stdout:7 2024-05-08T16:23:44.461 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-05-08T16:23:44.462 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:23:44.462 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T16:23:45.005 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:23:46.006 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:23:46.007 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:23:46.007 INFO:tasks.workunit.client.0.smithi172.stdout:8 2024-05-08T16:23:46.007 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-05-08T16:23:46.007 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:23:46.008 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T16:23:46.510 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:23:47.512 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:23:47.512 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:23:47.512 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-05-08T16:23:47.513 INFO:tasks.workunit.client.0.smithi172.stdout:9 2024-05-08T16:23:47.513 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:23:47.513 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T16:23:48.058 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:23:49.060 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:23:49.060 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:23:49.060 INFO:tasks.workunit.client.0.smithi172.stdout:10 2024-05-08T16:23:49.061 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-05-08T16:23:49.061 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:23:49.061 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T16:23:49.504 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:23:50.506 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:23:50.507 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:23:50.507 INFO:tasks.workunit.client.0.smithi172.stdout:11 2024-05-08T16:23:50.507 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-05-08T16:23:50.507 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:23:50.507 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T16:23:51.058 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:23:52.060 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:23:52.061 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:23:52.061 INFO:tasks.workunit.client.0.smithi172.stdout:12 2024-05-08T16:23:52.061 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-05-08T16:23:52.061 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:23:52.062 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T16:23:52.828 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:23:53.830 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:23:53.830 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:23:53.830 INFO:tasks.workunit.client.0.smithi172.stdout:13 2024-05-08T16:23:53.831 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-05-08T16:23:53.831 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:23:53.831 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T16:23:54.712 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:23:55.714 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:23:55.714 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:23:55.715 INFO:tasks.workunit.client.0.smithi172.stdout:14 2024-05-08T16:23:55.715 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-05-08T16:23:55.715 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:23:55.715 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T16:23:56.804 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:23:57.805 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:23:57.806 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:23:57.806 INFO:tasks.workunit.client.0.smithi172.stdout:15 2024-05-08T16:23:57.806 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-05-08T16:23:57.806 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:23:57.807 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T16:23:58.303 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:23:59.305 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:23:59.305 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:23:59.305 INFO:tasks.workunit.client.0.smithi172.stdout:16 2024-05-08T16:23:59.306 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-05-08T16:23:59.306 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:23:59.306 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T16:23:59.863 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:24:00.865 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:24:00.865 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:24:00.866 INFO:tasks.workunit.client.0.smithi172.stdout:17 2024-05-08T16:24:00.866 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-05-08T16:24:00.866 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:24:00.866 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T16:24:01.939 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:24:02.940 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:24:02.941 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:24:02.941 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-05-08T16:24:02.941 INFO:tasks.workunit.client.0.smithi172.stdout:18 2024-05-08T16:24:02.941 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:24:02.941 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T16:24:03.669 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:24:04.671 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:24:04.671 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:24:04.671 INFO:tasks.workunit.client.0.smithi172.stdout:19 2024-05-08T16:24:04.672 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-05-08T16:24:04.672 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:24:04.672 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T16:24:05.381 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:24:06.383 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:24:06.383 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:24:06.384 INFO:tasks.workunit.client.0.smithi172.stdout:20 2024-05-08T16:24:06.384 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-05-08T16:24:06.384 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:24:06.384 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T16:24:07.164 INFO:tasks.workunit.client.0.smithi172.stdout:osd.1 up in weight 1 up_from 13 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/2020169112,v1:127.0.0.1:6811/2020169112] [v2:127.0.0.1:6812/2020169112,v1:127.0.0.1:6813/2020169112] exists,up 44cea6e4-14d4-45e8-8eed-6bbcdcfa4563 2024-05-08T16:24:07.167 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-08T16:24:07.167 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-08T16:24:07.168 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-08T16:24:07.168 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-05-08T16:24:07.168 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 2 2024-05-08T16:24:07.168 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-05-08T16:24:07.168 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-05-08T16:24:07.168 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-05-08T16:24:07.168 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-05-08T16:24:07.168 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/2 2024-05-08T16:24:07.168 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=4457b0f1-028b-482b-8ee3-7ee05861e6cc --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-05-08T16:24:07.168 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-08T16:24:07.168 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-05-08T16:24:07.168 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-08T16:24:07.168 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/2' 2024-05-08T16:24:07.169 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/2/journal' 2024-05-08T16:24:07.169 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-05-08T16:24:07.169 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-05-08T16:24:07.169 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-05-08T16:24:07.169 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-05-08T16:24:07.169 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-08T16:24:07.169 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-08T16:24:07.169 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-08T16:24:07.169 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T16:24:07.169 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81110 2024-05-08T16:24:07.169 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.81110/$cluster-$name.asok' 2024-05-08T16:24:07.169 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.81110/$cluster-$name.asok' 2024-05-08T16:24:07.169 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-05-08T16:24:07.169 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-05-08T16:24:07.170 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-05-08T16:24:07.170 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-05-08T16:24:07.170 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-05-08T16:24:07.170 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-08T16:24:07.170 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-08T16:24:07.170 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-08T16:24:07.170 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-08T16:24:07.170 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-05-08T16:24:07.170 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-05-08T16:24:07.170 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/2 2024-05-08T16:24:07.171 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-05-08T16:24:07.171 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=078a47a1-3b00-45e4-bb1f-922301111710 2024-05-08T16:24:07.171 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 078a47a1-3b00-45e4-bb1f-922301111710' 2024-05-08T16:24:07.171 INFO:tasks.workunit.client.0.smithi172.stdout:add osd2 078a47a1-3b00-45e4-bb1f-922301111710 2024-05-08T16:24:07.171 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-05-08T16:24:07.193 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAnpztmts/iCxAAVhiAgrAcgh8+wr01jZJSLw== 2024-05-08T16:24:07.193 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAnpztmts/iCxAAVhiAgrAcgh8+wr01jZJSLw=="}' 2024-05-08T16:24:07.193 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 078a47a1-3b00-45e4-bb1f-922301111710 -i td/test-erasure-eio/2/new.json 2024-05-08T16:24:07.995 INFO:tasks.workunit.client.0.smithi172.stdout:2 2024-05-08T16:24:08.012 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/2/new.json 2024-05-08T16:24:08.013 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=4457b0f1-028b-482b-8ee3-7ee05861e6cc --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81110/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAnpztmts/iCxAAVhiAgrAcgh8+wr01jZJSLw== --osd-uuid 078a47a1-3b00-45e4-bb1f-922301111710 2024-05-08T16:24:08.047 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:24:08.048+0000 7fe2d44617c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:24:08.049 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:24:08.052+0000 7fe2d44617c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:24:08.050 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:24:08.052+0000 7fe2d44617c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:24:08.050 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:24:08.052+0000 7fe2d44617c0 -1 bluestore(td/test-erasure-eio/2/block) _read_bdev_label failed to open td/test-erasure-eio/2/block: (2) No such file or directory 2024-05-08T16:24:08.050 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:24:08.052+0000 7fe2d44617c0 -1 bluestore(td/test-erasure-eio/2/block) _read_bdev_label failed to open td/test-erasure-eio/2/block: (2) No such file or directory 2024-05-08T16:24:08.051 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:24:08.052+0000 7fe2d44617c0 -1 bluestore(td/test-erasure-eio/2/block) _read_bdev_label failed to open td/test-erasure-eio/2/block: (2) No such file or directory 2024-05-08T16:24:08.191 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:24:08.196+0000 7fe2d44617c0 -1 bluestore(td/test-erasure-eio/2) _read_fsid unparsable uuid 2024-05-08T16:24:15.565 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/2/keyring 2024-05-08T16:24:15.565 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-05-08T16:24:15.566 INFO:tasks.workunit.client.0.smithi172.stdout:adding osd2 key to auth repository 2024-05-08T16:24:15.567 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-05-08T16:24:15.567 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-05-08T16:24:16.122 INFO:tasks.workunit.client.0.smithi172.stdout:start osd.2 2024-05-08T16:24:16.123 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-05-08T16:24:16.123 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-05-08T16:24:16.124 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=4457b0f1-028b-482b-8ee3-7ee05861e6cc --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81110/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-05-08T16:24:16.124 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-05-08T16:24:16.124 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-05-08T16:24:16.155 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:24:16.156+0000 7f364bd547c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:24:16.157 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:24:16.160+0000 7f364bd547c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:24:16.159 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:24:16.160+0000 7f364bd547c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:24:16.538 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-05-08T16:24:16.539 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-08T16:24:16.539 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-05-08T16:24:16.539 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-08T16:24:16.539 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-08T16:24:16.539 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:24:16.539 INFO:tasks.workunit.client.0.smithi172.stdout:0 2024-05-08T16:24:16.540 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-08T16:24:16.540 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:24:16.540 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T16:24:16.722 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:24:16.724+0000 7f364bd547c0 -1 Falling back to public interface 2024-05-08T16:24:16.985 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:24:17.986 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:24:17.986 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:24:17.987 INFO:tasks.workunit.client.0.smithi172.stdout:1 2024-05-08T16:24:17.987 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-08T16:24:17.987 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:24:17.987 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T16:24:18.411 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:24:19.413 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:24:19.413 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:24:19.414 INFO:tasks.workunit.client.0.smithi172.stdout:2 2024-05-08T16:24:19.414 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-08T16:24:19.414 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:24:19.414 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T16:24:19.636 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:24:19.636+0000 7f364bd547c0 -1 osd.2 0 log_to_monitors true 2024-05-08T16:24:19.853 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:24:20.855 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:24:20.855 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:24:20.856 INFO:tasks.workunit.client.0.smithi172.stdout:3 2024-05-08T16:24:20.856 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-05-08T16:24:20.856 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:24:20.856 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T16:24:21.301 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:24:22.303 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:24:22.303 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:24:22.303 INFO:tasks.workunit.client.0.smithi172.stdout:4 2024-05-08T16:24:22.304 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-05-08T16:24:22.304 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:24:22.304 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T16:24:22.923 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:24:23.925 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:24:23.926 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:24:23.926 INFO:tasks.workunit.client.0.smithi172.stdout:5 2024-05-08T16:24:23.926 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-05-08T16:24:23.926 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:24:23.927 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T16:24:24.637 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:24:25.638 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:24:25.639 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:24:25.639 INFO:tasks.workunit.client.0.smithi172.stdout:6 2024-05-08T16:24:25.639 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-05-08T16:24:25.639 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:24:25.640 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T16:24:26.328 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:24:27.330 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:24:27.330 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:24:27.331 INFO:tasks.workunit.client.0.smithi172.stdout:7 2024-05-08T16:24:27.331 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-05-08T16:24:27.331 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:24:27.331 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T16:24:28.137 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:24:29.139 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:24:29.139 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:24:29.139 INFO:tasks.workunit.client.0.smithi172.stdout:8 2024-05-08T16:24:29.140 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-05-08T16:24:29.140 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:24:29.140 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T16:24:29.927 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:24:30.929 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:24:30.929 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:24:30.929 INFO:tasks.workunit.client.0.smithi172.stdout:9 2024-05-08T16:24:30.930 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-05-08T16:24:30.930 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:24:30.930 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T16:24:31.392 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:24:32.393 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:24:32.393 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:24:32.394 INFO:tasks.workunit.client.0.smithi172.stdout:10 2024-05-08T16:24:32.394 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-05-08T16:24:32.394 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:24:32.394 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T16:24:32.888 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:24:33.889 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:24:33.890 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:24:33.890 INFO:tasks.workunit.client.0.smithi172.stdout:11 2024-05-08T16:24:33.891 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-05-08T16:24:33.891 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:24:33.891 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T16:24:34.460 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:24:35.462 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:24:35.462 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:24:35.462 INFO:tasks.workunit.client.0.smithi172.stdout:12 2024-05-08T16:24:35.463 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-05-08T16:24:35.463 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:24:35.463 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T16:24:36.052 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:24:37.054 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:24:37.054 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:24:37.054 INFO:tasks.workunit.client.0.smithi172.stdout:13 2024-05-08T16:24:37.054 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-05-08T16:24:37.055 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:24:37.055 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T16:24:37.727 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:24:38.729 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:24:38.730 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:24:38.730 INFO:tasks.workunit.client.0.smithi172.stdout:14 2024-05-08T16:24:38.730 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-05-08T16:24:38.730 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:24:38.730 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T16:24:39.556 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:24:40.557 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:24:40.558 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:24:40.558 INFO:tasks.workunit.client.0.smithi172.stdout:15 2024-05-08T16:24:40.558 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-05-08T16:24:40.558 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:24:40.559 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T16:24:41.285 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:24:42.287 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:24:42.287 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:24:42.287 INFO:tasks.workunit.client.0.smithi172.stdout:16 2024-05-08T16:24:42.288 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-05-08T16:24:42.288 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:24:42.288 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T16:24:43.095 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:24:44.096 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:24:44.097 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:24:44.097 INFO:tasks.workunit.client.0.smithi172.stdout:17 2024-05-08T16:24:44.097 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-05-08T16:24:44.098 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:24:44.098 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T16:24:44.884 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:24:45.885 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:24:45.885 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:24:45.886 INFO:tasks.workunit.client.0.smithi172.stdout:18 2024-05-08T16:24:45.886 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-05-08T16:24:45.886 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:24:45.887 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T16:24:46.399 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:24:47.401 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:24:47.401 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:24:47.402 INFO:tasks.workunit.client.0.smithi172.stdout:19 2024-05-08T16:24:47.402 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-05-08T16:24:47.402 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:24:47.402 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T16:24:48.304 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:24:49.306 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:24:49.306 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:24:49.306 INFO:tasks.workunit.client.0.smithi172.stdout:20 2024-05-08T16:24:49.307 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-05-08T16:24:49.307 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:24:49.307 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T16:24:49.978 INFO:tasks.workunit.client.0.smithi172.stdout:osd.2 up in weight 1 up_from 19 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/2886273035,v1:127.0.0.1:6819/2886273035] [v2:127.0.0.1:6820/2886273035,v1:127.0.0.1:6821/2886273035] exists,up 078a47a1-3b00-45e4-bb1f-922301111710 2024-05-08T16:24:49.985 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-08T16:24:49.986 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-08T16:24:49.986 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-08T16:24:49.986 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-05-08T16:24:49.986 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 3 2024-05-08T16:24:49.986 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-05-08T16:24:49.986 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-05-08T16:24:49.986 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2024-05-08T16:24:49.987 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-05-08T16:24:49.987 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/3 2024-05-08T16:24:49.987 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=4457b0f1-028b-482b-8ee3-7ee05861e6cc --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-05-08T16:24:49.987 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-08T16:24:49.987 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-05-08T16:24:49.987 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-08T16:24:49.987 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/3' 2024-05-08T16:24:49.987 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/3/journal' 2024-05-08T16:24:49.988 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-05-08T16:24:49.988 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-05-08T16:24:49.988 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-05-08T16:24:49.988 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-05-08T16:24:49.988 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-08T16:24:49.988 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-08T16:24:49.988 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-08T16:24:49.988 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T16:24:49.989 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81110 2024-05-08T16:24:49.989 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.81110/$cluster-$name.asok' 2024-05-08T16:24:49.989 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.81110/$cluster-$name.asok' 2024-05-08T16:24:49.989 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-05-08T16:24:49.989 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-05-08T16:24:49.989 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-05-08T16:24:49.989 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-05-08T16:24:49.989 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-05-08T16:24:49.990 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-08T16:24:49.990 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-08T16:24:49.990 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-08T16:24:49.990 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-08T16:24:49.990 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-05-08T16:24:49.990 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-05-08T16:24:49.990 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/3 2024-05-08T16:24:49.990 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-05-08T16:24:49.990 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=7add20e4-33cb-4a93-939b-b30e7a4a4106 2024-05-08T16:24:49.991 INFO:tasks.workunit.client.0.smithi172.stdout:add osd3 7add20e4-33cb-4a93-939b-b30e7a4a4106 2024-05-08T16:24:49.991 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 7add20e4-33cb-4a93-939b-b30e7a4a4106' 2024-05-08T16:24:49.992 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-05-08T16:24:50.004 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBSpztmjUCaABAAqnP8ev855RWWH37G2UJz2Q== 2024-05-08T16:24:50.004 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBSpztmjUCaABAAqnP8ev855RWWH37G2UJz2Q=="}' 2024-05-08T16:24:50.004 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 7add20e4-33cb-4a93-939b-b30e7a4a4106 -i td/test-erasure-eio/3/new.json 2024-05-08T16:24:51.076 INFO:tasks.workunit.client.0.smithi172.stdout:3 2024-05-08T16:24:51.097 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/3/new.json 2024-05-08T16:24:51.098 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=4457b0f1-028b-482b-8ee3-7ee05861e6cc --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81110/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBSpztmjUCaABAAqnP8ev855RWWH37G2UJz2Q== --osd-uuid 7add20e4-33cb-4a93-939b-b30e7a4a4106 2024-05-08T16:24:51.133 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:24:51.139+0000 7efee1d6d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:24:51.253 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:24:51.259+0000 7efee1d6d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:24:51.255 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:24:51.259+0000 7efee1d6d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:24:51.255 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:24:51.259+0000 7efee1d6d7c0 -1 bluestore(td/test-erasure-eio/3/block) _read_bdev_label failed to open td/test-erasure-eio/3/block: (2) No such file or directory 2024-05-08T16:24:51.255 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:24:51.259+0000 7efee1d6d7c0 -1 bluestore(td/test-erasure-eio/3/block) _read_bdev_label failed to open td/test-erasure-eio/3/block: (2) No such file or directory 2024-05-08T16:24:51.255 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:24:51.259+0000 7efee1d6d7c0 -1 bluestore(td/test-erasure-eio/3/block) _read_bdev_label failed to open td/test-erasure-eio/3/block: (2) No such file or directory 2024-05-08T16:24:51.432 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:24:51.435+0000 7efee1d6d7c0 -1 bluestore(td/test-erasure-eio/3) _read_fsid unparsable uuid 2024-05-08T16:24:58.529 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/3/keyring 2024-05-08T16:24:58.530 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-05-08T16:24:58.531 INFO:tasks.workunit.client.0.smithi172.stdout:adding osd3 key to auth repository 2024-05-08T16:24:58.531 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2024-05-08T16:24:58.531 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-05-08T16:24:59.092 INFO:tasks.workunit.client.0.smithi172.stdout:start osd.3 2024-05-08T16:24:59.094 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2024-05-08T16:24:59.094 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=4457b0f1-028b-482b-8ee3-7ee05861e6cc --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81110/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-05-08T16:24:59.094 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-05-08T16:24:59.094 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-05-08T16:24:59.095 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-05-08T16:24:59.123 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:24:59.127+0000 7f55905577c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:24:59.124 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:24:59.131+0000 7f55905577c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:24:59.126 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:24:59.131+0000 7f55905577c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:24:59.503 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2024-05-08T16:24:59.503 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-08T16:24:59.503 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-05-08T16:24:59.503 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-08T16:24:59.503 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-08T16:24:59.503 INFO:tasks.workunit.client.0.smithi172.stdout:0 2024-05-08T16:24:59.504 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:24:59.504 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-08T16:24:59.504 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:24:59.504 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T16:24:59.682 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:24:59.687+0000 7f55905577c0 -1 Falling back to public interface 2024-05-08T16:24:59.974 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:25:00.975 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:25:00.976 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:25:00.976 INFO:tasks.workunit.client.0.smithi172.stdout:1 2024-05-08T16:25:00.976 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-08T16:25:00.977 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:25:00.977 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T16:25:01.381 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:25:02.382 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:25:02.383 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:25:02.383 INFO:tasks.workunit.client.0.smithi172.stdout:2 2024-05-08T16:25:02.383 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-08T16:25:02.384 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:25:02.384 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T16:25:02.422 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:25:02.427+0000 7f55905577c0 -1 osd.3 0 log_to_monitors true 2024-05-08T16:25:02.937 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:25:03.939 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:25:03.939 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:25:03.940 INFO:tasks.workunit.client.0.smithi172.stdout:3 2024-05-08T16:25:03.940 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-05-08T16:25:03.940 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:25:03.940 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T16:25:04.694 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:25:05.696 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:25:05.696 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:25:05.696 INFO:tasks.workunit.client.0.smithi172.stdout:4 2024-05-08T16:25:05.697 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-05-08T16:25:05.697 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:25:05.697 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T16:25:06.277 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:25:07.278 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:25:07.278 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:25:07.278 INFO:tasks.workunit.client.0.smithi172.stdout:5 2024-05-08T16:25:07.279 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-05-08T16:25:07.279 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:25:07.279 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T16:25:07.864 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:25:08.865 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:25:08.866 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:25:08.866 INFO:tasks.workunit.client.0.smithi172.stdout:6 2024-05-08T16:25:08.866 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-05-08T16:25:08.867 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:25:08.867 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T16:25:09.414 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:25:10.416 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:25:10.416 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:25:10.416 INFO:tasks.workunit.client.0.smithi172.stdout:7 2024-05-08T16:25:10.417 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-05-08T16:25:10.417 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:25:10.417 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T16:25:11.045 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:25:12.047 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:25:12.047 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:25:12.048 INFO:tasks.workunit.client.0.smithi172.stdout:8 2024-05-08T16:25:12.048 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-05-08T16:25:12.048 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:25:12.048 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T16:25:12.588 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:25:13.590 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:25:13.590 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:25:13.590 INFO:tasks.workunit.client.0.smithi172.stdout:9 2024-05-08T16:25:13.591 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-05-08T16:25:13.591 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:25:13.591 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T16:25:14.241 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:25:15.246 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:25:15.246 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:25:15.246 INFO:tasks.workunit.client.0.smithi172.stdout:10 2024-05-08T16:25:15.247 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-05-08T16:25:15.247 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:25:15.247 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T16:25:15.671 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:25:16.673 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:25:16.673 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:25:16.674 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-05-08T16:25:16.674 INFO:tasks.workunit.client.0.smithi172.stdout:11 2024-05-08T16:25:16.674 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:25:16.674 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T16:25:17.155 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:25:18.157 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:25:18.157 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:25:18.157 INFO:tasks.workunit.client.0.smithi172.stdout:12 2024-05-08T16:25:18.158 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-05-08T16:25:18.158 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:25:18.158 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T16:25:18.700 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:25:19.701 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:25:19.701 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:25:19.701 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-05-08T16:25:19.702 INFO:tasks.workunit.client.0.smithi172.stdout:13 2024-05-08T16:25:19.702 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:25:19.702 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T16:25:20.154 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:25:21.156 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:25:21.156 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:25:21.157 INFO:tasks.workunit.client.0.smithi172.stdout:14 2024-05-08T16:25:21.157 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-05-08T16:25:21.158 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:25:21.158 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T16:25:22.238 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:25:23.240 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:25:23.240 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:25:23.240 INFO:tasks.workunit.client.0.smithi172.stdout:15 2024-05-08T16:25:23.241 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-05-08T16:25:23.241 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:25:23.241 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T16:25:24.128 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:25:25.130 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:25:25.130 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:25:25.130 INFO:tasks.workunit.client.0.smithi172.stdout:16 2024-05-08T16:25:25.131 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-05-08T16:25:25.131 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:25:25.131 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T16:25:26.196 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:25:27.198 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:25:27.199 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:25:27.199 INFO:tasks.workunit.client.0.smithi172.stdout:17 2024-05-08T16:25:27.199 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-05-08T16:25:27.200 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:25:27.200 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T16:25:28.318 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:25:29.319 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:25:29.320 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:25:29.320 INFO:tasks.workunit.client.0.smithi172.stdout:18 2024-05-08T16:25:29.320 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-05-08T16:25:29.321 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:25:29.321 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T16:25:30.205 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:25:31.207 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:25:31.207 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:25:31.208 INFO:tasks.workunit.client.0.smithi172.stdout:19 2024-05-08T16:25:31.208 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-05-08T16:25:31.208 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:25:31.208 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T16:25:32.048 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:25:33.050 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:25:33.050 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:25:33.050 INFO:tasks.workunit.client.0.smithi172.stdout:20 2024-05-08T16:25:33.051 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-05-08T16:25:33.051 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:25:33.051 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T16:25:33.736 INFO:tasks.workunit.client.0.smithi172.stdout:osd.3 up in weight 1 up_from 27 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/1060156486,v1:127.0.0.1:6827/1060156486] [v2:127.0.0.1:6828/1060156486,v1:127.0.0.1:6829/1060156486] exists,up 7add20e4-33cb-4a93-939b-b30e7a4a4106 2024-05-08T16:25:33.740 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-08T16:25:33.740 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-08T16:25:33.740 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-08T16:25:33.740 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-05-08T16:25:33.740 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 4 2024-05-08T16:25:33.740 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-05-08T16:25:33.740 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-05-08T16:25:33.740 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=4 2024-05-08T16:25:33.741 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-05-08T16:25:33.741 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/4 2024-05-08T16:25:33.741 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=4457b0f1-028b-482b-8ee3-7ee05861e6cc --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-05-08T16:25:33.741 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-08T16:25:33.741 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-05-08T16:25:33.741 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-08T16:25:33.741 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/4' 2024-05-08T16:25:33.741 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/4/journal' 2024-05-08T16:25:33.741 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-05-08T16:25:33.741 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-05-08T16:25:33.741 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-05-08T16:25:33.741 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-05-08T16:25:33.741 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-08T16:25:33.741 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-08T16:25:33.742 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-08T16:25:33.742 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T16:25:33.742 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81110 2024-05-08T16:25:33.742 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.81110/$cluster-$name.asok' 2024-05-08T16:25:33.742 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.81110/$cluster-$name.asok' 2024-05-08T16:25:33.742 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-05-08T16:25:33.742 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-05-08T16:25:33.742 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-05-08T16:25:33.742 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-05-08T16:25:33.742 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-05-08T16:25:33.742 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-08T16:25:33.742 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-08T16:25:33.742 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-08T16:25:33.742 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-08T16:25:33.742 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-05-08T16:25:33.743 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-05-08T16:25:33.743 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/4 2024-05-08T16:25:33.743 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-05-08T16:25:33.743 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=3ea802cd-f4ef-4d89-ac55-1c2ae44a8af3 2024-05-08T16:25:33.743 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd4 3ea802cd-f4ef-4d89-ac55-1c2ae44a8af3' 2024-05-08T16:25:33.744 INFO:tasks.workunit.client.0.smithi172.stdout:add osd4 3ea802cd-f4ef-4d89-ac55-1c2ae44a8af3 2024-05-08T16:25:33.744 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-05-08T16:25:33.764 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQB9pztmiEDsLRAA9m7V80Z7RaaUw7RpUKUUnQ== 2024-05-08T16:25:33.764 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQB9pztmiEDsLRAA9m7V80Z7RaaUw7RpUKUUnQ=="}' 2024-05-08T16:25:33.764 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 3ea802cd-f4ef-4d89-ac55-1c2ae44a8af3 -i td/test-erasure-eio/4/new.json 2024-05-08T16:25:34.986 INFO:tasks.workunit.client.0.smithi172.stdout:4 2024-05-08T16:25:35.006 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/4/new.json 2024-05-08T16:25:35.007 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 4 --fsid=4457b0f1-028b-482b-8ee3-7ee05861e6cc --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/4 --osd-journal=td/test-erasure-eio/4/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81110/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQB9pztmiEDsLRAA9m7V80Z7RaaUw7RpUKUUnQ== --osd-uuid 3ea802cd-f4ef-4d89-ac55-1c2ae44a8af3 2024-05-08T16:25:35.043 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:25:35.046+0000 7feec4bff7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:25:35.067 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:25:35.070+0000 7feec4bff7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:25:35.069 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:25:35.074+0000 7feec4bff7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:25:35.069 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:25:35.074+0000 7feec4bff7c0 -1 bluestore(td/test-erasure-eio/4/block) _read_bdev_label failed to open td/test-erasure-eio/4/block: (2) No such file or directory 2024-05-08T16:25:35.069 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:25:35.074+0000 7feec4bff7c0 -1 bluestore(td/test-erasure-eio/4/block) _read_bdev_label failed to open td/test-erasure-eio/4/block: (2) No such file or directory 2024-05-08T16:25:35.069 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:25:35.074+0000 7feec4bff7c0 -1 bluestore(td/test-erasure-eio/4/block) _read_bdev_label failed to open td/test-erasure-eio/4/block: (2) No such file or directory 2024-05-08T16:25:35.236 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:25:35.238+0000 7feec4bff7c0 -1 bluestore(td/test-erasure-eio/4) _read_fsid unparsable uuid 2024-05-08T16:25:43.157 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/4/keyring 2024-05-08T16:25:43.157 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-05-08T16:25:43.159 INFO:tasks.workunit.client.0.smithi172.stdout:adding osd4 key to auth repository 2024-05-08T16:25:43.159 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd4 key to auth repository 2024-05-08T16:25:43.159 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-05-08T16:25:43.722 INFO:tasks.workunit.client.0.smithi172.stdout:start osd.4 2024-05-08T16:25:43.723 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.4 2024-05-08T16:25:43.723 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 4 --fsid=4457b0f1-028b-482b-8ee3-7ee05861e6cc --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/4 --osd-journal=td/test-erasure-eio/4/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81110/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-05-08T16:25:43.723 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-05-08T16:25:43.723 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-05-08T16:25:43.723 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-05-08T16:25:43.754 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:25:43.758+0000 7f178eb007c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:25:43.756 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:25:43.762+0000 7f178eb007c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:25:43.757 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:25:43.762+0000 7f178eb007c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:25:44.141 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 4 2024-05-08T16:25:44.141 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-08T16:25:44.141 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=4 2024-05-08T16:25:44.142 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-08T16:25:44.142 INFO:tasks.workunit.client.0.smithi172.stdout:0 2024-05-08T16:25:44.142 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-08T16:25:44.143 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:25:44.143 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-08T16:25:44.143 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:25:44.143 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-08T16:25:44.295 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:25:44.298+0000 7f178eb007c0 -1 Falling back to public interface 2024-05-08T16:25:44.538 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:25:45.540 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:25:45.540 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:25:45.540 INFO:tasks.workunit.client.0.smithi172.stdout:1 2024-05-08T16:25:45.541 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-08T16:25:45.541 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:25:45.541 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-08T16:25:45.960 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:25:46.946 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:25:46.950+0000 7f178eb007c0 -1 osd.4 0 log_to_monitors true 2024-05-08T16:25:46.961 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:25:46.961 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:25:46.962 INFO:tasks.workunit.client.0.smithi172.stdout:2 2024-05-08T16:25:46.962 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-08T16:25:46.962 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-08T16:25:46.962 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:25:47.389 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:25:48.391 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:25:48.391 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:25:48.392 INFO:tasks.workunit.client.0.smithi172.stdout:3 2024-05-08T16:25:48.392 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-05-08T16:25:48.392 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:25:48.392 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-08T16:25:49.463 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:25:50.464 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:25:50.464 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:25:50.464 INFO:tasks.workunit.client.0.smithi172.stdout:4 2024-05-08T16:25:50.465 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-05-08T16:25:50.465 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:25:50.465 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-08T16:25:51.124 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:25:52.126 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:25:52.126 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:25:52.127 INFO:tasks.workunit.client.0.smithi172.stdout:5 2024-05-08T16:25:52.127 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-05-08T16:25:52.128 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:25:52.128 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-08T16:25:52.789 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:25:53.791 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:25:53.791 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:25:53.791 INFO:tasks.workunit.client.0.smithi172.stdout:6 2024-05-08T16:25:53.791 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-05-08T16:25:53.792 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:25:53.792 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-08T16:25:54.419 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:25:55.421 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:25:55.421 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:25:55.421 INFO:tasks.workunit.client.0.smithi172.stdout:7 2024-05-08T16:25:55.422 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-05-08T16:25:55.422 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:25:55.422 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-08T16:25:56.063 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:25:57.065 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:25:57.065 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:25:57.066 INFO:tasks.workunit.client.0.smithi172.stdout:8 2024-05-08T16:25:57.066 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-05-08T16:25:57.066 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:25:57.067 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-08T16:25:57.523 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:25:58.524 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:25:58.524 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:25:58.524 INFO:tasks.workunit.client.0.smithi172.stdout:9 2024-05-08T16:25:58.525 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-05-08T16:25:58.525 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:25:58.525 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-08T16:25:59.088 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:26:00.090 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:26:00.090 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:26:00.090 INFO:tasks.workunit.client.0.smithi172.stdout:10 2024-05-08T16:26:00.091 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-05-08T16:26:00.091 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:26:00.091 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-08T16:26:00.589 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:26:01.591 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:26:01.591 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:26:01.592 INFO:tasks.workunit.client.0.smithi172.stdout:11 2024-05-08T16:26:01.592 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-05-08T16:26:01.592 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:26:01.592 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-08T16:26:02.152 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:26:03.153 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:26:03.154 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:26:03.154 INFO:tasks.workunit.client.0.smithi172.stdout:12 2024-05-08T16:26:03.155 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-05-08T16:26:03.155 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:26:03.155 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-08T16:26:03.588 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:26:04.589 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:26:04.590 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:26:04.590 INFO:tasks.workunit.client.0.smithi172.stdout:13 2024-05-08T16:26:04.590 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-05-08T16:26:04.590 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:26:04.590 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-08T16:26:05.246 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:26:06.248 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:26:06.248 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:26:06.248 INFO:tasks.workunit.client.0.smithi172.stdout:14 2024-05-08T16:26:06.249 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-05-08T16:26:06.249 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:26:06.250 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-08T16:26:07.001 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:26:08.003 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:26:08.003 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:26:08.004 INFO:tasks.workunit.client.0.smithi172.stdout:15 2024-05-08T16:26:08.004 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-05-08T16:26:08.004 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:26:08.005 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-08T16:26:09.265 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:26:10.270 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:26:10.270 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:26:10.271 INFO:tasks.workunit.client.0.smithi172.stdout:16 2024-05-08T16:26:10.271 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-05-08T16:26:10.271 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:26:10.271 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-08T16:26:11.194 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:26:12.196 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:26:12.196 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:26:12.197 INFO:tasks.workunit.client.0.smithi172.stdout:17 2024-05-08T16:26:12.197 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-05-08T16:26:12.197 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:26:12.197 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-08T16:26:13.297 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:26:14.299 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:26:14.299 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:26:14.299 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-05-08T16:26:14.300 INFO:tasks.workunit.client.0.smithi172.stdout:18 2024-05-08T16:26:14.300 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:26:14.300 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-08T16:26:15.124 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:26:16.125 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:26:16.125 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:26:16.126 INFO:tasks.workunit.client.0.smithi172.stdout:19 2024-05-08T16:26:16.126 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-05-08T16:26:16.127 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:26:16.127 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-08T16:26:17.250 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:26:18.252 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:26:18.252 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:26:18.253 INFO:tasks.workunit.client.0.smithi172.stdout:20 2024-05-08T16:26:18.253 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-05-08T16:26:18.253 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:26:18.253 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-08T16:26:18.942 INFO:tasks.workunit.client.0.smithi172.stdout:osd.4 up in weight 1 up_from 37 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/488008837,v1:127.0.0.1:6835/488008837] [v2:127.0.0.1:6836/488008837,v1:127.0.0.1:6837/488008837] exists,up 3ea802cd-f4ef-4d89-ac55-1c2ae44a8af3 2024-05-08T16:26:18.947 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-08T16:26:18.947 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-08T16:26:18.947 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-08T16:26:18.947 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-05-08T16:26:18.947 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 5 2024-05-08T16:26:18.947 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-05-08T16:26:18.947 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-05-08T16:26:18.947 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=5 2024-05-08T16:26:18.947 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-05-08T16:26:18.947 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/5 2024-05-08T16:26:18.947 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=4457b0f1-028b-482b-8ee3-7ee05861e6cc --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-05-08T16:26:18.947 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-08T16:26:18.947 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-05-08T16:26:18.948 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-08T16:26:18.948 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/5' 2024-05-08T16:26:18.948 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/5/journal' 2024-05-08T16:26:18.948 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-05-08T16:26:18.948 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-05-08T16:26:18.948 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-05-08T16:26:18.949 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-05-08T16:26:18.949 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-08T16:26:18.949 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-08T16:26:18.949 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-08T16:26:18.949 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T16:26:18.949 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81110 2024-05-08T16:26:18.949 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.81110/$cluster-$name.asok' 2024-05-08T16:26:18.949 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.81110/$cluster-$name.asok' 2024-05-08T16:26:18.949 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-05-08T16:26:18.949 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-05-08T16:26:18.950 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-05-08T16:26:18.950 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-05-08T16:26:18.950 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-05-08T16:26:18.950 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-08T16:26:18.950 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-08T16:26:18.950 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-08T16:26:18.950 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-08T16:26:18.950 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-05-08T16:26:18.950 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-05-08T16:26:18.950 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/5 2024-05-08T16:26:18.951 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-05-08T16:26:18.951 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=fcd23b71-e2c6-48d9-a26b-7297e873a52f 2024-05-08T16:26:18.951 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd5 fcd23b71-e2c6-48d9-a26b-7297e873a52f' 2024-05-08T16:26:18.951 INFO:tasks.workunit.client.0.smithi172.stdout:add osd5 fcd23b71-e2c6-48d9-a26b-7297e873a52f 2024-05-08T16:26:18.951 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-05-08T16:26:18.971 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCqpztmfio3OhAAcb22F0AQ9JgTxLV1tIwB8Q== 2024-05-08T16:26:18.971 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCqpztmfio3OhAAcb22F0AQ9JgTxLV1tIwB8Q=="}' 2024-05-08T16:26:18.971 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new fcd23b71-e2c6-48d9-a26b-7297e873a52f -i td/test-erasure-eio/5/new.json 2024-05-08T16:26:20.021 INFO:tasks.workunit.client.0.smithi172.stdout:5 2024-05-08T16:26:20.043 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/5/new.json 2024-05-08T16:26:20.044 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 5 --fsid=4457b0f1-028b-482b-8ee3-7ee05861e6cc --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/5 --osd-journal=td/test-erasure-eio/5/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81110/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCqpztmfio3OhAAcb22F0AQ9JgTxLV1tIwB8Q== --osd-uuid fcd23b71-e2c6-48d9-a26b-7297e873a52f 2024-05-08T16:26:20.082 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:26:20.085+0000 7f9320e707c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:26:20.084 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:26:20.085+0000 7f9320e707c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:26:20.085 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:26:20.089+0000 7f9320e707c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:26:20.086 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:26:20.089+0000 7f9320e707c0 -1 bluestore(td/test-erasure-eio/5/block) _read_bdev_label failed to open td/test-erasure-eio/5/block: (2) No such file or directory 2024-05-08T16:26:20.086 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:26:20.089+0000 7f9320e707c0 -1 bluestore(td/test-erasure-eio/5/block) _read_bdev_label failed to open td/test-erasure-eio/5/block: (2) No such file or directory 2024-05-08T16:26:20.086 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:26:20.089+0000 7f9320e707c0 -1 bluestore(td/test-erasure-eio/5/block) _read_bdev_label failed to open td/test-erasure-eio/5/block: (2) No such file or directory 2024-05-08T16:26:20.220 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:26:20.221+0000 7f9320e707c0 -1 bluestore(td/test-erasure-eio/5) _read_fsid unparsable uuid 2024-05-08T16:26:27.631 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/5/keyring 2024-05-08T16:26:27.632 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-05-08T16:26:27.633 INFO:tasks.workunit.client.0.smithi172.stdout:adding osd5 key to auth repository 2024-05-08T16:26:27.633 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd5 key to auth repository 2024-05-08T16:26:27.634 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/5/keyring auth add osd.5 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-05-08T16:26:28.187 INFO:tasks.workunit.client.0.smithi172.stdout:start osd.5 2024-05-08T16:26:28.188 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.5 2024-05-08T16:26:28.188 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-05-08T16:26:28.188 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-05-08T16:26:28.188 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-05-08T16:26:28.188 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 5 --fsid=4457b0f1-028b-482b-8ee3-7ee05861e6cc --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/5 --osd-journal=td/test-erasure-eio/5/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81110/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-05-08T16:26:28.219 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:26:28.221+0000 7fc4153be7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:26:28.229 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:26:28.233+0000 7fc4153be7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:26:28.230 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:26:28.233+0000 7fc4153be7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:26:28.608 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 5 2024-05-08T16:26:28.608 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-08T16:26:28.608 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=5 2024-05-08T16:26:28.608 INFO:tasks.workunit.client.0.smithi172.stdout:0 2024-05-08T16:26:28.609 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-08T16:26:28.609 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-08T16:26:28.609 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:26:28.609 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-08T16:26:28.609 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:26:28.609 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-05-08T16:26:28.793 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:26:28.797+0000 7fc4153be7c0 -1 Falling back to public interface 2024-05-08T16:26:29.022 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:26:30.024 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:26:30.024 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:26:30.024 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-08T16:26:30.024 INFO:tasks.workunit.client.0.smithi172.stdout:1 2024-05-08T16:26:30.025 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:26:30.025 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-05-08T16:26:30.420 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:26:31.421 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:26:31.421 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:26:31.422 INFO:tasks.workunit.client.0.smithi172.stdout:2 2024-05-08T16:26:31.422 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-08T16:26:31.422 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:26:31.422 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-05-08T16:26:31.459 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:26:31.460+0000 7fc4153be7c0 -1 osd.5 0 log_to_monitors true 2024-05-08T16:26:31.971 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:26:32.972 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:26:32.972 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:26:32.973 INFO:tasks.workunit.client.0.smithi172.stdout:3 2024-05-08T16:26:32.973 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-05-08T16:26:32.973 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:26:32.973 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-05-08T16:26:33.443 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:26:34.445 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:26:34.445 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:26:34.445 INFO:tasks.workunit.client.0.smithi172.stdout:4 2024-05-08T16:26:34.446 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-05-08T16:26:34.446 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:26:34.446 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-05-08T16:26:35.063 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:26:36.064 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:26:36.065 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:26:36.065 INFO:tasks.workunit.client.0.smithi172.stdout:5 2024-05-08T16:26:36.065 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-05-08T16:26:36.066 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:26:36.066 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-05-08T16:26:36.637 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:26:37.639 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:26:37.639 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:26:37.640 INFO:tasks.workunit.client.0.smithi172.stdout:6 2024-05-08T16:26:37.641 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-05-08T16:26:37.641 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:26:37.641 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-05-08T16:26:38.150 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:26:39.151 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:26:39.151 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:26:39.152 INFO:tasks.workunit.client.0.smithi172.stdout:7 2024-05-08T16:26:39.152 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-05-08T16:26:39.152 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:26:39.152 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-05-08T16:26:39.582 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:26:40.584 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:26:40.584 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:26:40.584 INFO:tasks.workunit.client.0.smithi172.stdout:8 2024-05-08T16:26:40.585 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-05-08T16:26:40.585 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:26:40.585 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-05-08T16:26:41.010 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:26:42.012 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:26:42.012 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:26:42.012 INFO:tasks.workunit.client.0.smithi172.stdout:9 2024-05-08T16:26:42.013 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-05-08T16:26:42.013 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:26:42.013 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-05-08T16:26:42.426 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:26:43.428 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:26:43.428 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:26:43.429 INFO:tasks.workunit.client.0.smithi172.stdout:10 2024-05-08T16:26:43.429 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-05-08T16:26:43.429 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:26:43.429 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-05-08T16:26:44.216 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:26:45.218 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:26:45.218 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:26:45.219 INFO:tasks.workunit.client.0.smithi172.stdout:11 2024-05-08T16:26:45.219 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-05-08T16:26:45.219 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:26:45.220 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-05-08T16:26:45.950 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:26:46.951 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:26:46.951 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:26:46.952 INFO:tasks.workunit.client.0.smithi172.stdout:12 2024-05-08T16:26:46.952 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-05-08T16:26:46.952 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:26:46.952 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-05-08T16:26:47.425 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:26:48.426 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:26:48.427 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:26:48.427 INFO:tasks.workunit.client.0.smithi172.stdout:13 2024-05-08T16:26:48.427 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-05-08T16:26:48.428 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:26:48.428 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-05-08T16:26:48.901 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:26:49.902 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:26:49.903 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:26:49.903 INFO:tasks.workunit.client.0.smithi172.stdout:14 2024-05-08T16:26:49.903 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-05-08T16:26:49.904 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:26:49.904 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-05-08T16:26:50.424 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:26:51.426 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:26:51.426 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:26:51.426 INFO:tasks.workunit.client.0.smithi172.stdout:15 2024-05-08T16:26:51.427 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-05-08T16:26:51.427 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:26:51.427 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-05-08T16:26:52.578 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:26:53.580 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:26:53.580 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:26:53.580 INFO:tasks.workunit.client.0.smithi172.stdout:16 2024-05-08T16:26:53.581 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-05-08T16:26:53.581 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:26:53.581 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-05-08T16:26:54.699 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:26:55.701 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:26:55.701 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:26:55.701 INFO:tasks.workunit.client.0.smithi172.stdout:17 2024-05-08T16:26:55.702 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-05-08T16:26:55.702 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:26:55.702 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-05-08T16:26:56.993 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:26:57.995 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:26:57.995 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:26:57.995 INFO:tasks.workunit.client.0.smithi172.stdout:18 2024-05-08T16:26:57.996 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-05-08T16:26:57.996 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:26:57.996 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-05-08T16:26:58.494 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:26:59.496 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:26:59.496 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:26:59.496 INFO:tasks.workunit.client.0.smithi172.stdout:19 2024-05-08T16:26:59.497 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-05-08T16:26:59.497 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:26:59.497 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-05-08T16:27:00.494 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:27:01.496 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:27:01.496 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:27:01.496 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-05-08T16:27:01.497 INFO:tasks.workunit.client.0.smithi172.stdout:20 2024-05-08T16:27:01.497 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:27:01.497 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-05-08T16:27:02.286 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:27:03.288 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:27:03.288 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:27:03.288 INFO:tasks.workunit.client.0.smithi172.stdout:21 2024-05-08T16:27:03.289 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-05-08T16:27:03.289 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:27:03.289 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-05-08T16:27:03.717 INFO:tasks.workunit.client.0.smithi172.stdout:osd.5 up in weight 1 up_from 47 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6842/1889540355,v1:127.0.0.1:6843/1889540355] [v2:127.0.0.1:6844/1889540355,v1:127.0.0.1:6845/1889540355] exists,up fcd23b71-e2c6-48d9-a26b-7297e873a52f 2024-05-08T16:27:03.720 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-08T16:27:03.720 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-08T16:27:03.720 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-08T16:27:03.720 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-05-08T16:27:03.720 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 6 2024-05-08T16:27:03.720 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-05-08T16:27:03.720 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-05-08T16:27:03.720 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=6 2024-05-08T16:27:03.720 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-05-08T16:27:03.720 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/6 2024-05-08T16:27:03.720 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=4457b0f1-028b-482b-8ee3-7ee05861e6cc --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-05-08T16:27:03.721 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-08T16:27:03.721 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-05-08T16:27:03.721 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-08T16:27:03.721 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/6' 2024-05-08T16:27:03.721 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/6/journal' 2024-05-08T16:27:03.721 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-05-08T16:27:03.721 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-05-08T16:27:03.721 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-05-08T16:27:03.721 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-05-08T16:27:03.721 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-08T16:27:03.721 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-08T16:27:03.721 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-08T16:27:03.721 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T16:27:03.721 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81110 2024-05-08T16:27:03.721 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.81110/$cluster-$name.asok' 2024-05-08T16:27:03.723 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.81110/$cluster-$name.asok' 2024-05-08T16:27:03.723 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-05-08T16:27:03.723 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-05-08T16:27:03.723 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-05-08T16:27:03.724 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-05-08T16:27:03.724 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-05-08T16:27:03.724 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-08T16:27:03.724 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-08T16:27:03.724 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-08T16:27:03.724 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-08T16:27:03.724 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-05-08T16:27:03.724 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-05-08T16:27:03.724 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/6 2024-05-08T16:27:03.724 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-05-08T16:27:03.724 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=26be4227-a1c9-494f-a624-7b35bc6e9c4f 2024-05-08T16:27:03.724 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd6 26be4227-a1c9-494f-a624-7b35bc6e9c4f' 2024-05-08T16:27:03.725 INFO:tasks.workunit.client.0.smithi172.stdout:add osd6 26be4227-a1c9-494f-a624-7b35bc6e9c4f 2024-05-08T16:27:03.725 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-05-08T16:27:03.746 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDXpztmVb/PLBAAeQWdGo4k9PO7lWG0N8I3jg== 2024-05-08T16:27:03.746 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDXpztmVb/PLBAAeQWdGo4k9PO7lWG0N8I3jg=="}' 2024-05-08T16:27:03.746 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 26be4227-a1c9-494f-a624-7b35bc6e9c4f -i td/test-erasure-eio/6/new.json 2024-05-08T16:27:04.844 INFO:tasks.workunit.client.0.smithi172.stdout:6 2024-05-08T16:27:04.860 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/6/new.json 2024-05-08T16:27:04.860 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 6 --fsid=4457b0f1-028b-482b-8ee3-7ee05861e6cc --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/6 --osd-journal=td/test-erasure-eio/6/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81110/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDXpztmVb/PLBAAeQWdGo4k9PO7lWG0N8I3jg== --osd-uuid 26be4227-a1c9-494f-a624-7b35bc6e9c4f 2024-05-08T16:27:04.898 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:27:04.896+0000 7f81d16767c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:27:04.934 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:27:04.936+0000 7f81d16767c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:27:04.936 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:27:04.936+0000 7f81d16767c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:27:04.936 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:27:04.940+0000 7f81d16767c0 -1 bluestore(td/test-erasure-eio/6/block) _read_bdev_label failed to open td/test-erasure-eio/6/block: (2) No such file or directory 2024-05-08T16:27:04.936 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:27:04.940+0000 7f81d16767c0 -1 bluestore(td/test-erasure-eio/6/block) _read_bdev_label failed to open td/test-erasure-eio/6/block: (2) No such file or directory 2024-05-08T16:27:04.936 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:27:04.940+0000 7f81d16767c0 -1 bluestore(td/test-erasure-eio/6/block) _read_bdev_label failed to open td/test-erasure-eio/6/block: (2) No such file or directory 2024-05-08T16:27:04.974 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:27:04.976+0000 7f81d16767c0 -1 bluestore(td/test-erasure-eio/6) _read_fsid unparsable uuid 2024-05-08T16:27:11.549 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/6/keyring 2024-05-08T16:27:11.550 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-05-08T16:27:11.550 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd6 key to auth repository 2024-05-08T16:27:11.551 INFO:tasks.workunit.client.0.smithi172.stdout:adding osd6 key to auth repository 2024-05-08T16:27:11.551 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/6/keyring auth add osd.6 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-05-08T16:27:12.257 INFO:tasks.workunit.client.0.smithi172.stdout:start osd.6 2024-05-08T16:27:12.258 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.6 2024-05-08T16:27:12.258 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 6 --fsid=4457b0f1-028b-482b-8ee3-7ee05861e6cc --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/6 --osd-journal=td/test-erasure-eio/6/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81110/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-05-08T16:27:12.258 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-05-08T16:27:12.258 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-05-08T16:27:12.258 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-05-08T16:27:12.289 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:27:12.292+0000 7f3c723e47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:27:12.314 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:27:12.316+0000 7f3c723e47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:27:12.315 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:27:12.316+0000 7f3c723e47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:27:12.721 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 6 2024-05-08T16:27:12.721 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-08T16:27:12.721 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=6 2024-05-08T16:27:12.721 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-08T16:27:12.721 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-08T16:27:12.721 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:27:12.721 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-08T16:27:12.722 INFO:tasks.workunit.client.0.smithi172.stdout:0 2024-05-08T16:27:12.722 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:27:12.722 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-05-08T16:27:12.863 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:27:12.864+0000 7f3c723e47c0 -1 Falling back to public interface 2024-05-08T16:27:13.174 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:27:14.176 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:27:14.176 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:27:14.176 INFO:tasks.workunit.client.0.smithi172.stdout:1 2024-05-08T16:27:14.177 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-08T16:27:14.177 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:27:14.177 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-05-08T16:27:14.590 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:27:15.372 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:27:15.371+0000 7f3c723e47c0 -1 osd.6 0 log_to_monitors true 2024-05-08T16:27:15.592 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:27:15.592 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:27:15.593 INFO:tasks.workunit.client.0.smithi172.stdout:2 2024-05-08T16:27:15.593 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-08T16:27:15.593 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:27:15.593 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-05-08T16:27:16.266 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:27:17.267 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:27:17.267 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:27:17.268 INFO:tasks.workunit.client.0.smithi172.stdout:3 2024-05-08T16:27:17.268 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-05-08T16:27:17.268 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:27:17.268 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-05-08T16:27:17.801 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:27:18.805 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:27:18.805 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:27:18.806 INFO:tasks.workunit.client.0.smithi172.stdout:4 2024-05-08T16:27:18.806 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-05-08T16:27:18.806 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:27:18.806 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-05-08T16:27:19.561 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:27:20.562 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:27:20.563 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:27:20.563 INFO:tasks.workunit.client.0.smithi172.stdout:5 2024-05-08T16:27:20.563 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-05-08T16:27:20.563 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:27:20.563 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-05-08T16:27:21.056 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:27:22.058 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:27:22.058 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:27:22.059 INFO:tasks.workunit.client.0.smithi172.stdout:6 2024-05-08T16:27:22.059 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-05-08T16:27:22.059 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:27:22.059 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-05-08T16:27:22.512 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:27:23.514 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:27:23.514 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:27:23.515 INFO:tasks.workunit.client.0.smithi172.stdout:7 2024-05-08T16:27:23.515 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-05-08T16:27:23.515 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:27:23.515 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-05-08T16:27:24.168 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:27:25.170 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:27:25.171 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:27:25.171 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-05-08T16:27:25.171 INFO:tasks.workunit.client.0.smithi172.stdout:8 2024-05-08T16:27:25.171 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:27:25.171 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-05-08T16:27:25.593 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:27:26.594 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:27:26.594 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:27:26.595 INFO:tasks.workunit.client.0.smithi172.stdout:9 2024-05-08T16:27:26.595 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-05-08T16:27:26.595 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:27:26.595 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-05-08T16:27:27.043 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:27:28.044 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:27:28.045 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:27:28.045 INFO:tasks.workunit.client.0.smithi172.stdout:10 2024-05-08T16:27:28.045 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-05-08T16:27:28.046 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:27:28.046 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-05-08T16:27:28.504 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:27:29.506 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:27:29.506 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:27:29.506 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-05-08T16:27:29.506 INFO:tasks.workunit.client.0.smithi172.stdout:11 2024-05-08T16:27:29.507 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:27:29.507 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-05-08T16:27:29.946 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:27:30.948 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:27:30.948 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:27:30.949 INFO:tasks.workunit.client.0.smithi172.stdout:12 2024-05-08T16:27:30.949 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-05-08T16:27:30.949 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:27:30.949 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-05-08T16:27:31.447 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:27:32.449 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:27:32.449 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:27:32.450 INFO:tasks.workunit.client.0.smithi172.stdout:13 2024-05-08T16:27:32.450 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-05-08T16:27:32.450 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:27:32.450 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-05-08T16:27:33.538 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:27:34.540 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:27:34.541 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:27:34.541 INFO:tasks.workunit.client.0.smithi172.stdout:14 2024-05-08T16:27:34.541 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-05-08T16:27:34.542 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:27:34.542 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-05-08T16:27:35.625 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:27:36.626 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:27:36.627 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:27:36.627 INFO:tasks.workunit.client.0.smithi172.stdout:15 2024-05-08T16:27:36.627 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-05-08T16:27:36.628 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:27:36.628 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-05-08T16:27:37.523 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:27:38.525 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:27:38.526 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:27:38.526 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-05-08T16:27:38.526 INFO:tasks.workunit.client.0.smithi172.stdout:16 2024-05-08T16:27:38.526 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:27:38.526 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-05-08T16:27:39.543 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:27:40.545 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:27:40.545 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:27:40.546 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-05-08T16:27:40.546 INFO:tasks.workunit.client.0.smithi172.stdout:17 2024-05-08T16:27:40.546 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:27:40.546 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-05-08T16:27:41.587 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:27:42.589 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:27:42.589 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:27:42.589 INFO:tasks.workunit.client.0.smithi172.stdout:18 2024-05-08T16:27:42.590 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-05-08T16:27:42.590 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:27:42.590 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-05-08T16:27:43.121 INFO:tasks.workunit.client.0.smithi172.stdout:osd.6 up in weight 1 up_from 56 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6850/3543354842,v1:127.0.0.1:6851/3543354842] [v2:127.0.0.1:6852/3543354842,v1:127.0.0.1:6853/3543354842] exists,up 26be4227-a1c9-494f-a624-7b35bc6e9c4f 2024-05-08T16:27:43.122 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-08T16:27:43.122 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-08T16:27:43.122 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-08T16:27:43.122 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: get_asok_path osd.0 2024-05-08T16:27:43.122 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-05-08T16:27:43.122 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-05-08T16:27:43.123 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-08T16:27:43.123 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T16:27:43.123 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81110 2024-05-08T16:27:43.123 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.81110/ceph-osd.0.asok 2024-05-08T16:27:43.124 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: CEPH_ARGS= 2024-05-08T16:27:43.124 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: ceph --admin-daemon /tmp/ceph-asok.81110/ceph-osd.0.asok log flush 2024-05-08T16:27:43.236 INFO:tasks.workunit.client.0.smithi172.stdout:{} 2024-05-08T16:27:43.248 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:56: setup_osds: grep 'load: jerasure.*lrc' td/test-erasure-eio/osd.0.log 2024-05-08T16:27:43.256 INFO:tasks.workunit.client.0.smithi172.stdout:2024-05-08T16:22:51.562+0000 7f8507dc67c0 0 load: jerasure load: lrc 2024-05-08T16:27:43.258 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:364: TEST_ec_object_attr_read_error: local poolname=pool-jerasure 2024-05-08T16:27:43.258 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:365: TEST_ec_object_attr_read_error: create_erasure_coded_pool pool-jerasure 3 2 2024-05-08T16:27:43.258 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:67: create_erasure_coded_pool: local poolname=pool-jerasure 2024-05-08T16:27:43.258 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:68: create_erasure_coded_pool: shift 2024-05-08T16:27:43.258 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:69: create_erasure_coded_pool: local k=3 2024-05-08T16:27:43.258 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:70: create_erasure_coded_pool: shift 2024-05-08T16:27:43.258 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:71: create_erasure_coded_pool: local m=2 2024-05-08T16:27:43.258 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:72: create_erasure_coded_pool: shift 2024-05-08T16:27:43.258 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:74: create_erasure_coded_pool: ceph osd erasure-code-profile set myprofile plugin=jerasure k=3 m=2 crush-failure-domain=osd 2024-05-08T16:27:44.049 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:78: create_erasure_coded_pool: create_pool pool-jerasure 1 1 erasure myprofile 2024-05-08T16:27:44.049 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create pool-jerasure 1 1 erasure myprofile 2024-05-08T16:27:45.648 INFO:tasks.workunit.client.0.smithi172.stderr:pool 'pool-jerasure' already exists 2024-05-08T16:27:45.671 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-05-08T16:27:46.673 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:80: create_erasure_coded_pool: wait_for_clean 2024-05-08T16:27:46.673 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-05-08T16:27:46.673 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-05-08T16:27:46.673 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-05-08T16:27:46.674 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-05-08T16:27:46.674 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-05-08T16:27:46.675 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-05-08T16:27:46.675 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-05-08T16:27:46.675 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-05-08T16:27:46.675 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-05-08T16:27:46.796 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-05-08T16:27:46.796 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-05-08T16:27:46.796 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-05-08T16:27:46.796 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-05-08T16:27:46.796 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-05-08T16:27:46.796 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-05-08T16:27:47.382 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-05-08T16:27:47.382 INFO:tasks.workunit.client.0.smithi172.stderr:1 2024-05-08T16:27:47.383 INFO:tasks.workunit.client.0.smithi172.stderr:2 2024-05-08T16:27:47.383 INFO:tasks.workunit.client.0.smithi172.stderr:3 2024-05-08T16:27:47.383 INFO:tasks.workunit.client.0.smithi172.stderr:4 2024-05-08T16:27:47.383 INFO:tasks.workunit.client.0.smithi172.stderr:5 2024-05-08T16:27:47.383 INFO:tasks.workunit.client.0.smithi172.stderr:6' 2024-05-08T16:27:47.383 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-05-08T16:27:47.383 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T16:27:47.383 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-05-08T16:27:47.581 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=25769803835 2024-05-08T16:27:47.581 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 25769803835 2024-05-08T16:27:47.581 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803835' 2024-05-08T16:27:47.581 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T16:27:47.581 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-05-08T16:27:47.736 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=55834574900 2024-05-08T16:27:47.736 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 55834574900 2024-05-08T16:27:47.736 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803835 1-55834574900' 2024-05-08T16:27:47.736 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T16:27:47.736 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-05-08T16:27:47.983 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=81604378670 2024-05-08T16:27:47.983 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 81604378670 2024-05-08T16:27:47.983 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803835 1-55834574900 2-81604378670' 2024-05-08T16:27:47.983 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T16:27:47.983 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-05-08T16:27:48.144 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=115964117023 2024-05-08T16:27:48.145 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 115964117023 2024-05-08T16:27:48.145 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803835 1-55834574900 2-81604378670 3-115964117023' 2024-05-08T16:27:48.145 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T16:27:48.145 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-05-08T16:27:48.374 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=158913789976 2024-05-08T16:27:48.374 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 158913789976 2024-05-08T16:27:48.374 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803835 1-55834574900 2-81604378670 3-115964117023 4-158913789976' 2024-05-08T16:27:48.374 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T16:27:48.375 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-05-08T16:27:48.559 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=201863462931 2024-05-08T16:27:48.560 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 201863462931 2024-05-08T16:27:48.560 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803835 1-55834574900 2-81604378670 3-115964117023 4-158913789976 5-201863462931' 2024-05-08T16:27:48.560 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T16:27:48.560 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-05-08T16:27:48.713 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=240518168583 2024-05-08T16:27:48.713 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 240518168583 2024-05-08T16:27:48.713 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803835 1-55834574900 2-81604378670 3-115964117023 4-158913789976 5-201863462931 6-240518168583' 2024-05-08T16:27:48.713 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-08T16:27:48.713 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-25769803835 2024-05-08T16:27:48.713 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-08T16:27:48.714 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-05-08T16:27:48.715 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-25769803835 2024-05-08T16:27:48.715 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-08T16:27:48.716 INFO:tasks.workunit.client.0.smithi172.stdout:waiting osd.0 seq 25769803835 2024-05-08T16:27:48.717 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=25769803835 2024-05-08T16:27:48.717 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 25769803835' 2024-05-08T16:27:48.717 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-05-08T16:27:49.159 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 25769803833 -lt 25769803835 2024-05-08T16:27:49.159 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-05-08T16:27:50.160 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-05-08T16:27:50.161 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-05-08T16:27:50.577 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 25769803836 -lt 25769803835 2024-05-08T16:27:50.578 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-08T16:27:50.578 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-55834574900 2024-05-08T16:27:50.579 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-08T16:27:50.581 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-05-08T16:27:50.581 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-55834574900 2024-05-08T16:27:50.581 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-08T16:27:50.582 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=55834574900 2024-05-08T16:27:50.583 INFO:tasks.workunit.client.0.smithi172.stdout:waiting osd.1 seq 55834574900 2024-05-08T16:27:50.583 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 55834574900' 2024-05-08T16:27:50.583 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-05-08T16:27:50.975 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 55834574900 -lt 55834574900 2024-05-08T16:27:50.975 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-08T16:27:50.976 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-08T16:27:50.976 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-81604378670 2024-05-08T16:27:50.977 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-05-08T16:27:50.978 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-81604378670 2024-05-08T16:27:50.978 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-08T16:27:50.979 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=81604378670 2024-05-08T16:27:50.979 INFO:tasks.workunit.client.0.smithi172.stdout:waiting osd.2 seq 81604378670 2024-05-08T16:27:50.980 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 81604378670' 2024-05-08T16:27:50.980 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-05-08T16:27:51.382 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 81604378670 -lt 81604378670 2024-05-08T16:27:51.383 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-08T16:27:51.383 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-115964117023 2024-05-08T16:27:51.383 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-08T16:27:51.385 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-05-08T16:27:51.386 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-115964117023 2024-05-08T16:27:51.386 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-08T16:27:51.387 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=115964117023 2024-05-08T16:27:51.387 INFO:tasks.workunit.client.0.smithi172.stdout:waiting osd.3 seq 115964117023 2024-05-08T16:27:51.387 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 115964117023' 2024-05-08T16:27:51.387 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-05-08T16:27:51.840 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 115964117024 -lt 115964117023 2024-05-08T16:27:51.840 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-08T16:27:51.840 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-158913789976 2024-05-08T16:27:51.841 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-08T16:27:51.842 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-05-08T16:27:51.842 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-158913789976 2024-05-08T16:27:51.842 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-08T16:27:51.843 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=158913789976 2024-05-08T16:27:51.844 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 158913789976' 2024-05-08T16:27:51.844 INFO:tasks.workunit.client.0.smithi172.stdout:waiting osd.4 seq 158913789976 2024-05-08T16:27:51.844 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-05-08T16:27:52.303 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 158913789977 -lt 158913789976 2024-05-08T16:27:52.304 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-08T16:27:52.304 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-201863462931 2024-05-08T16:27:52.305 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-08T16:27:52.306 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-05-08T16:27:52.307 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-201863462931 2024-05-08T16:27:52.307 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-08T16:27:52.308 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=201863462931 2024-05-08T16:27:52.308 INFO:tasks.workunit.client.0.smithi172.stdout:waiting osd.5 seq 201863462931 2024-05-08T16:27:52.309 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 201863462931' 2024-05-08T16:27:52.309 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-05-08T16:27:52.741 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 201863462931 -lt 201863462931 2024-05-08T16:27:52.741 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-08T16:27:52.742 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-08T16:27:52.742 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-240518168583 2024-05-08T16:27:52.743 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-05-08T16:27:52.744 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-240518168583 2024-05-08T16:27:52.744 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-08T16:27:52.745 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=240518168583 2024-05-08T16:27:52.745 INFO:tasks.workunit.client.0.smithi172.stdout:waiting osd.6 seq 240518168583 2024-05-08T16:27:52.746 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 240518168583' 2024-05-08T16:27:52.746 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-05-08T16:27:53.193 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 240518168583 -lt 240518168583 2024-05-08T16:27:53.194 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-05-08T16:27:53.194 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-05-08T16:27:53.194 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-05-08T16:27:53.750 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-05-08T16:27:53.751 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-05-08T16:27:53.751 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-05-08T16:27:53.751 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-05-08T16:27:53.752 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-05-08T16:27:53.752 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-05-08T16:27:53.752 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-05-08T16:27:53.752 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-05-08T16:27:54.152 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-05-08T16:27:54.153 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-05-08T16:27:54.153 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-05-08T16:27:54.153 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-05-08T16:27:54.659 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-05-08T16:27:54.659 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-05-08T16:27:54.659 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-05-08T16:27:54.659 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:367: TEST_ec_object_attr_read_error: get_primary pool-jerasure myobject 2024-05-08T16:27:54.659 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=pool-jerasure 2024-05-08T16:27:54.659 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=myobject 2024-05-08T16:27:54.660 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map pool-jerasure myobject 2024-05-08T16:27:54.660 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-05-08T16:27:55.109 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:367: TEST_ec_object_attr_read_error: local primary_osd=3 2024-05-08T16:27:55.109 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:369: TEST_ec_object_attr_read_error: kill_daemons td/test-erasure-eio TERM osd.3 2024-05-08T16:27:55.109 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-08T16:27:55.110 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-08T16:27:55.110 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-08T16:27:55.110 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-08T16:27:55.110 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-08T16:27:55.420 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-08T16:27:55.420 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:372: TEST_ec_object_attr_read_error: rados_put td/test-erasure-eio pool-jerasure myobject 2024-05-08T16:27:55.420 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:90: rados_put: local dir=td/test-erasure-eio 2024-05-08T16:27:55.420 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:91: rados_put: local poolname=pool-jerasure 2024-05-08T16:27:55.420 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:92: rados_put: local objname=myobject 2024-05-08T16:27:55.420 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-05-08T16:27:55.420 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 AAA 2024-05-08T16:27:55.420 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-05-08T16:27:55.420 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 BBB 2024-05-08T16:27:55.420 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-05-08T16:27:55.421 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 CCCC 2024-05-08T16:27:55.421 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-05-08T16:27:55.421 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 DDDD 2024-05-08T16:27:55.421 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:100: rados_put: rados --pool pool-jerasure put myobject td/test-erasure-eio/ORIGINAL 2024-05-08T16:27:57.485 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:375: TEST_ec_object_attr_read_error: inject_eio ec mdata pool-jerasure myobject td/test-erasure-eio 1 2024-05-08T16:27:57.485 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2394: inject_eio: local pooltype=ec 2024-05-08T16:27:57.485 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2395: inject_eio: shift 2024-05-08T16:27:57.485 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2396: inject_eio: local which=mdata 2024-05-08T16:27:57.485 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2397: inject_eio: shift 2024-05-08T16:27:57.485 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2398: inject_eio: local poolname=pool-jerasure 2024-05-08T16:27:57.485 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2399: inject_eio: shift 2024-05-08T16:27:57.485 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2400: inject_eio: local objname=myobject 2024-05-08T16:27:57.485 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2401: inject_eio: shift 2024-05-08T16:27:57.485 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2402: inject_eio: local dir=td/test-erasure-eio 2024-05-08T16:27:57.486 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2403: inject_eio: shift 2024-05-08T16:27:57.486 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2404: inject_eio: local shard_id=1 2024-05-08T16:27:57.486 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2405: inject_eio: shift 2024-05-08T16:27:57.486 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: get_osds pool-jerasure myobject 2024-05-08T16:27:57.486 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2024-05-08T16:27:57.486 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=myobject 2024-05-08T16:27:57.486 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure myobject 2024-05-08T16:27:57.486 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-05-08T16:27:57.923 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=2147483647 2024-05-08T16:27:57.923 INFO:tasks.workunit.client.0.smithi172.stderr:5 2024-05-08T16:27:57.923 INFO:tasks.workunit.client.0.smithi172.stderr:1 2024-05-08T16:27:57.924 INFO:tasks.workunit.client.0.smithi172.stderr:0 2024-05-08T16:27:57.924 INFO:tasks.workunit.client.0.smithi172.stderr:6' 2024-05-08T16:27:57.924 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 2147483647 5 1 0 6 2024-05-08T16:27:57.924 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: initial_osds=('2147483647' '5' '1' '0' '6') 2024-05-08T16:27:57.924 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: local -a initial_osds 2024-05-08T16:27:57.924 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2408: inject_eio: local osd_id=5 2024-05-08T16:27:57.924 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2409: inject_eio: '[' ec '!=' ec ']' 2024-05-08T16:27:57.925 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: cat td/test-erasure-eio/5/type 2024-05-08T16:27:57.925 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: type=bluestore 2024-05-08T16:27:57.925 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: set_config osd 5 bluestore_debug_inject_read_err true 2024-05-08T16:27:57.925 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-05-08T16:27:57.925 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=5 2024-05-08T16:27:57.925 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-05-08T16:27:57.925 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-05-08T16:27:57.926 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-05-08T16:27:57.926 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.5 2024-05-08T16:27:57.927 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2024-05-08T16:27:57.927 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2024-05-08T16:27:57.927 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-08T16:27:57.927 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T16:27:57.927 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81110 2024-05-08T16:27:57.927 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.81110/ceph-osd.5.asok 2024-05-08T16:27:57.927 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.81110/ceph-osd.5.asok config set bluestore_debug_inject_read_err true 2024-05-08T16:27:58.049 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-05-08T16:27:58.050 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local loop=0 2024-05-08T16:27:58.050 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: grep -q Invalid 2024-05-08T16:27:58.051 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: get_asok_path osd.5 2024-05-08T16:27:58.051 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.5 2024-05-08T16:27:58.051 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.5 ']' 2024-05-08T16:27:58.052 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-08T16:27:58.052 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T16:27:58.052 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81110 2024-05-08T16:27:58.052 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.81110/ceph-osd.5.asok 2024-05-08T16:27:58.052 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: CEPH_ARGS= 2024-05-08T16:27:58.052 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: ceph --admin-daemon /tmp/ceph-asok.81110/ceph-osd.5.asok injectmdataerr pool-jerasure myobject 1 2024-05-08T16:27:58.173 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:378: TEST_ec_object_attr_read_error: activate_osd td/test-erasure-eio 3 2024-05-08T16:27:58.174 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/test-erasure-eio 2024-05-08T16:27:58.174 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-05-08T16:27:58.174 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2024-05-08T16:27:58.174 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-05-08T16:27:58.174 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/test-erasure-eio/3 2024-05-08T16:27:58.174 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=4457b0f1-028b-482b-8ee3-7ee05861e6cc --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-05-08T16:27:58.174 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-08T16:27:58.174 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-05-08T16:27:58.174 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-08T16:27:58.174 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/3' 2024-05-08T16:27:58.174 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/3/journal' 2024-05-08T16:27:58.174 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-05-08T16:27:58.175 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-05-08T16:27:58.175 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-05-08T16:27:58.175 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-05-08T16:27:58.175 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-08T16:27:58.175 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-08T16:27:58.175 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-08T16:27:58.175 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T16:27:58.175 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81110 2024-05-08T16:27:58.176 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.81110/$cluster-$name.asok' 2024-05-08T16:27:58.176 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.81110/$cluster-$name.asok' 2024-05-08T16:27:58.177 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-05-08T16:27:58.177 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-05-08T16:27:58.177 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-05-08T16:27:58.177 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-08T16:27:58.177 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-08T16:27:58.177 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-08T16:27:58.177 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-08T16:27:58.177 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-05-08T16:27:58.177 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-05-08T16:27:58.177 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/test-erasure-eio/3 2024-05-08T16:27:58.178 INFO:tasks.workunit.client.0.smithi172.stdout:start osd.3 2024-05-08T16:27:58.178 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2024-05-08T16:27:58.179 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=4457b0f1-028b-482b-8ee3-7ee05861e6cc --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81110/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-05-08T16:27:58.179 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/test-erasure-eio/3/whoami 2024-05-08T16:27:58.179 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2024-05-08T16:27:58.179 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-05-08T16:27:58.179 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-05-08T16:27:58.179 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-05-08T16:27:58.213 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:27:58.219+0000 7fd067fae7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:27:58.215 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:27:58.219+0000 7fd067fae7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:27:58.216 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:27:58.223+0000 7fd067fae7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:27:58.601 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2024-05-08T16:27:58.601 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-08T16:27:58.601 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-05-08T16:27:58.601 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-08T16:27:58.602 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-08T16:27:58.602 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:27:58.602 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-08T16:27:58.602 INFO:tasks.workunit.client.0.smithi172.stdout:0 2024-05-08T16:27:58.602 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T16:27:58.602 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:27:58.773 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:27:58.779+0000 7fd067fae7c0 -1 Falling back to public interface 2024-05-08T16:27:59.047 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:28:00.049 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:28:00.049 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:28:00.049 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-08T16:28:00.050 INFO:tasks.workunit.client.0.smithi172.stdout:1 2024-05-08T16:28:00.050 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:28:00.050 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T16:28:00.453 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:28:01.456 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:28:01.456 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:28:01.456 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-08T16:28:01.456 INFO:tasks.workunit.client.0.smithi172.stdout:2 2024-05-08T16:28:01.457 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:28:01.457 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T16:28:01.882 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:28:01.999 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:28:02.003+0000 7fd067fae7c0 -1 osd.3 64 log_to_monitors true 2024-05-08T16:28:02.883 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:28:02.883 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:28:02.883 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-05-08T16:28:02.883 INFO:tasks.workunit.client.0.smithi172.stdout:3 2024-05-08T16:28:02.884 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:28:02.884 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T16:28:03.554 INFO:tasks.workunit.client.0.smithi172.stdout:osd.3 up in weight 1 up_from 68 up_thru 68 down_at 65 last_clean_interval [27,64) [v2:127.0.0.1:6826/912470406,v1:127.0.0.1:6827/912470406] [v2:127.0.0.1:6828/912470406,v1:127.0.0.1:6829/912470406] exists,up 7add20e4-33cb-4a93-939b-b30e7a4a4106 2024-05-08T16:28:03.555 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-08T16:28:03.555 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-08T16:28:03.556 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-08T16:28:03.556 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:381: TEST_ec_object_attr_read_error: wait_for_clean 2024-05-08T16:28:03.556 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-05-08T16:28:03.556 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-05-08T16:28:03.556 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-05-08T16:28:03.556 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-05-08T16:28:03.556 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-05-08T16:28:03.556 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-05-08T16:28:03.556 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-05-08T16:28:03.556 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-05-08T16:28:03.556 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-05-08T16:28:03.664 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-05-08T16:28:03.664 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-05-08T16:28:03.664 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-05-08T16:28:03.664 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-05-08T16:28:03.665 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-05-08T16:28:03.665 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-05-08T16:28:04.442 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-05-08T16:28:04.442 INFO:tasks.workunit.client.0.smithi172.stderr:1 2024-05-08T16:28:04.443 INFO:tasks.workunit.client.0.smithi172.stderr:2 2024-05-08T16:28:04.443 INFO:tasks.workunit.client.0.smithi172.stderr:3 2024-05-08T16:28:04.443 INFO:tasks.workunit.client.0.smithi172.stderr:4 2024-05-08T16:28:04.443 INFO:tasks.workunit.client.0.smithi172.stderr:5 2024-05-08T16:28:04.443 INFO:tasks.workunit.client.0.smithi172.stderr:6' 2024-05-08T16:28:04.443 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-05-08T16:28:04.443 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T16:28:04.443 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-05-08T16:28:04.707 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=25769803841 2024-05-08T16:28:04.707 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 25769803841 2024-05-08T16:28:04.707 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803841' 2024-05-08T16:28:04.707 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T16:28:04.707 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-05-08T16:28:04.878 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=55834574905 2024-05-08T16:28:04.879 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 55834574905 2024-05-08T16:28:04.879 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803841 1-55834574905' 2024-05-08T16:28:04.879 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T16:28:04.879 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-05-08T16:28:05.032 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=81604378675 2024-05-08T16:28:05.032 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 81604378675 2024-05-08T16:28:05.032 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803841 1-55834574905 2-81604378675' 2024-05-08T16:28:05.032 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T16:28:05.032 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-05-08T16:28:05.205 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=292057776131 2024-05-08T16:28:05.205 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 292057776131 2024-05-08T16:28:05.205 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803841 1-55834574905 2-81604378675 3-292057776131' 2024-05-08T16:28:05.205 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T16:28:05.205 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-05-08T16:28:05.356 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=158913789982 2024-05-08T16:28:05.357 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 158913789982 2024-05-08T16:28:05.357 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803841 1-55834574905 2-81604378675 3-292057776131 4-158913789982' 2024-05-08T16:28:05.357 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T16:28:05.357 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-05-08T16:28:05.516 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=201863462936 2024-05-08T16:28:05.516 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 201863462936 2024-05-08T16:28:05.516 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803841 1-55834574905 2-81604378675 3-292057776131 4-158913789982 5-201863462936' 2024-05-08T16:28:05.516 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T16:28:05.516 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-05-08T16:28:05.673 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=240518168588 2024-05-08T16:28:05.673 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 240518168588 2024-05-08T16:28:05.673 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803841 1-55834574905 2-81604378675 3-292057776131 4-158913789982 5-201863462936 6-240518168588' 2024-05-08T16:28:05.673 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-08T16:28:05.673 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-25769803841 2024-05-08T16:28:05.674 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-08T16:28:05.674 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-05-08T16:28:05.675 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-25769803841 2024-05-08T16:28:05.675 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-08T16:28:05.676 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=25769803841 2024-05-08T16:28:05.676 INFO:tasks.workunit.client.0.smithi172.stdout:waiting osd.0 seq 25769803841 2024-05-08T16:28:05.676 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 25769803841' 2024-05-08T16:28:05.676 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-05-08T16:28:06.110 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 25769803841 -lt 25769803841 2024-05-08T16:28:06.110 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-08T16:28:06.111 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-55834574905 2024-05-08T16:28:06.111 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-08T16:28:06.112 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-05-08T16:28:06.112 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-55834574905 2024-05-08T16:28:06.113 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-08T16:28:06.113 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=55834574905 2024-05-08T16:28:06.114 INFO:tasks.workunit.client.0.smithi172.stdout:waiting osd.1 seq 55834574905 2024-05-08T16:28:06.114 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 55834574905' 2024-05-08T16:28:06.114 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-05-08T16:28:06.568 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 55834574905 -lt 55834574905 2024-05-08T16:28:06.568 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-08T16:28:06.569 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-81604378675 2024-05-08T16:28:06.569 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-08T16:28:06.570 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-05-08T16:28:06.571 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-81604378675 2024-05-08T16:28:06.571 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-08T16:28:06.572 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=81604378675 2024-05-08T16:28:06.572 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 81604378675' 2024-05-08T16:28:06.572 INFO:tasks.workunit.client.0.smithi172.stdout:waiting osd.2 seq 81604378675 2024-05-08T16:28:06.573 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-05-08T16:28:07.027 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 81604378676 -lt 81604378675 2024-05-08T16:28:07.028 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-08T16:28:07.028 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-292057776131 2024-05-08T16:28:07.029 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-08T16:28:07.030 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-05-08T16:28:07.031 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-292057776131 2024-05-08T16:28:07.031 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-08T16:28:07.033 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=292057776131 2024-05-08T16:28:07.033 INFO:tasks.workunit.client.0.smithi172.stdout:waiting osd.3 seq 292057776131 2024-05-08T16:28:07.033 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 292057776131' 2024-05-08T16:28:07.033 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-05-08T16:28:07.444 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 292057776131 -lt 292057776131 2024-05-08T16:28:07.444 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-08T16:28:07.444 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-158913789982 2024-05-08T16:28:07.445 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-08T16:28:07.447 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-05-08T16:28:07.448 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-158913789982 2024-05-08T16:28:07.448 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-08T16:28:07.449 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=158913789982 2024-05-08T16:28:07.449 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 158913789982' 2024-05-08T16:28:07.449 INFO:tasks.workunit.client.0.smithi172.stdout:waiting osd.4 seq 158913789982 2024-05-08T16:28:07.450 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-05-08T16:28:07.911 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 158913789982 -lt 158913789982 2024-05-08T16:28:07.912 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-08T16:28:07.912 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-08T16:28:07.912 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-201863462936 2024-05-08T16:28:07.913 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-05-08T16:28:07.914 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-201863462936 2024-05-08T16:28:07.918 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-08T16:28:07.918 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=201863462936 2024-05-08T16:28:07.918 INFO:tasks.workunit.client.0.smithi172.stdout:waiting osd.5 seq 201863462936 2024-05-08T16:28:07.919 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 201863462936' 2024-05-08T16:28:07.919 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-05-08T16:28:08.346 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 201863462936 -lt 201863462936 2024-05-08T16:28:08.346 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-08T16:28:08.347 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-240518168588 2024-05-08T16:28:08.347 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-08T16:28:08.348 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-05-08T16:28:08.349 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-240518168588 2024-05-08T16:28:08.349 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-08T16:28:08.350 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=240518168588 2024-05-08T16:28:08.350 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 240518168588' 2024-05-08T16:28:08.350 INFO:tasks.workunit.client.0.smithi172.stdout:waiting osd.6 seq 240518168588 2024-05-08T16:28:08.351 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-05-08T16:28:08.818 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 240518168588 -lt 240518168588 2024-05-08T16:28:08.819 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-05-08T16:28:08.819 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-05-08T16:28:08.820 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-05-08T16:28:09.373 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-05-08T16:28:09.374 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-05-08T16:28:09.374 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-05-08T16:28:09.374 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-05-08T16:28:09.374 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-05-08T16:28:09.374 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-05-08T16:28:09.375 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-05-08T16:28:09.375 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-05-08T16:28:09.855 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-05-08T16:28:09.856 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-05-08T16:28:09.856 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-05-08T16:28:09.856 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-05-08T16:28:10.384 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-05-08T16:28:10.385 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-05-08T16:28:10.385 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-05-08T16:28:10.385 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:383: TEST_ec_object_attr_read_error: rados_get td/test-erasure-eio pool-jerasure myobject 2024-05-08T16:28:10.385 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2024-05-08T16:28:10.385 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2024-05-08T16:28:10.385 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=myobject 2024-05-08T16:28:10.385 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=ok 2024-05-08T16:28:10.385 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' ok = fail ']' 2024-05-08T16:28:10.385 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:120: rados_get: rados --pool pool-jerasure get myobject td/test-erasure-eio/COPY 2024-05-08T16:28:10.419 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:121: rados_get: diff td/test-erasure-eio/ORIGINAL td/test-erasure-eio/COPY 2024-05-08T16:28:10.420 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:122: rados_get: rm td/test-erasure-eio/COPY 2024-05-08T16:28:10.421 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:385: TEST_ec_object_attr_read_error: delete_erasure_coded_pool pool-jerasure 2024-05-08T16:28:10.421 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:84: delete_erasure_coded_pool: local poolname=pool-jerasure 2024-05-08T16:28:10.421 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:85: delete_erasure_coded_pool: ceph osd pool delete pool-jerasure pool-jerasure --yes-i-really-really-mean-it 2024-05-08T16:28:11.035 INFO:tasks.workunit.client.0.smithi172.stderr:pool 'pool-jerasure' does not exist 2024-05-08T16:28:11.051 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:86: delete_erasure_coded_pool: ceph osd erasure-code-profile rm myprofile 2024-05-08T16:28:11.780 INFO:tasks.workunit.client.0.smithi172.stderr:erasure-code-profile myprofile does not exist 2024-05-08T16:28:11.797 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:42: run: teardown td/test-erasure-eio 2024-05-08T16:28:11.797 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-eio 2024-05-08T16:28:11.797 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-05-08T16:28:11.797 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-eio KILL 2024-05-08T16:28:11.798 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-08T16:28:11.798 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-08T16:28:11.799 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-08T16:28:11.799 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-08T16:28:11.799 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-08T16:28:12.109 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-08T16:28:12.109 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-05-08T16:28:12.110 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-05-08T16:28:12.110 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-05-08T16:28:12.111 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-05-08T16:28:12.111 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-05-08T16:28:12.112 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-05-08T16:28:12.113 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-08T16:28:12.113 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-05-08T16:28:12.113 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-05-08T16:28:12.114 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-08T16:28:12.115 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-05-08T16:28:12.116 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-05-08T16:28:12.116 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-eio 2024-05-08T16:28:12.157 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-05-08T16:28:12.157 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T16:28:12.157 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81110 2024-05-08T16:28:12.157 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.81110 2024-05-08T16:28:12.159 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-05-08T16:28:12.159 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-05-08T16:28:12.159 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:32: run: for func in $funcs 2024-05-08T16:28:12.159 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:33: run: setup td/test-erasure-eio 2024-05-08T16:28:12.159 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/test-erasure-eio 2024-05-08T16:28:12.159 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/test-erasure-eio 2024-05-08T16:28:12.159 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-eio 2024-05-08T16:28:12.159 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-05-08T16:28:12.160 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-eio KILL 2024-05-08T16:28:12.160 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-08T16:28:12.160 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-08T16:28:12.160 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-08T16:28:12.160 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-08T16:28:12.160 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-08T16:28:12.161 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-08T16:28:12.162 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-05-08T16:28:12.162 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-05-08T16:28:12.163 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-05-08T16:28:12.164 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-05-08T16:28:12.164 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-05-08T16:28:12.164 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-05-08T16:28:12.165 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-08T16:28:12.165 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-05-08T16:28:12.166 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-05-08T16:28:12.166 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-08T16:28:12.167 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-05-08T16:28:12.168 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-05-08T16:28:12.168 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-eio 2024-05-08T16:28:12.170 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-05-08T16:28:12.170 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T16:28:12.170 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81110 2024-05-08T16:28:12.170 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.81110 2024-05-08T16:28:12.170 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-05-08T16:28:12.171 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-05-08T16:28:12.171 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/test-erasure-eio 2024-05-08T16:28:12.172 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-05-08T16:28:12.172 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T16:28:12.172 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81110 2024-05-08T16:28:12.172 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.81110 2024-05-08T16:28:12.174 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-05-08T16:28:12.175 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-05-08T16:28:12.175 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-05-08T16:28:12.175 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/test-erasure-eio 1' TERM HUP INT 2024-05-08T16:28:12.175 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:34: run: run_mon td/test-erasure-eio a 2024-05-08T16:28:12.175 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/test-erasure-eio 2024-05-08T16:28:12.175 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-05-08T16:28:12.175 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-05-08T16:28:12.175 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-05-08T16:28:12.175 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/test-erasure-eio/a 2024-05-08T16:28:12.175 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-eio/a --run-dir=td/test-erasure-eio 2024-05-08T16:28:12.599 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-05-08T16:28:12.600 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-08T16:28:12.600 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-08T16:28:12.600 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-08T16:28:12.600 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T16:28:12.600 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81110 2024-05-08T16:28:12.601 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.81110/$cluster-$name.asok' 2024-05-08T16:28:12.601 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-eio/a '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.81110/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-eio/log --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-05-08T16:28:12.834 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-05-08T16:28:12.835 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-05-08T16:28:12.835 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-05-08T16:28:12.835 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-05-08T16:28:12.836 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-05-08T16:28:12.836 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-05-08T16:28:12.836 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-05-08T16:28:12.836 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-05-08T16:28:12.837 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-05-08T16:28:12.837 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-08T16:28:12.837 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T16:28:12.837 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81110 2024-05-08T16:28:12.838 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.81110/ceph-mon.a.asok 2024-05-08T16:28:12.838 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-05-08T16:28:12.838 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.81110/ceph-mon.a.asok config get fsid 2024-05-08T16:28:12.949 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-05-08T16:28:12.949 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-05-08T16:28:12.949 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-05-08T16:28:12.949 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-05-08T16:28:12.951 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-05-08T16:28:12.951 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-05-08T16:28:12.951 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-05-08T16:28:12.951 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-05-08T16:28:12.951 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-08T16:28:12.951 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T16:28:12.951 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81110 2024-05-08T16:28:12.951 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.81110/ceph-mon.a.asok 2024-05-08T16:28:12.951 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-05-08T16:28:12.951 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.81110/ceph-mon.a.asok config get mon_host 2024-05-08T16:28:13.060 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:35: run: run_mgr td/test-erasure-eio x 2024-05-08T16:28:13.060 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/test-erasure-eio 2024-05-08T16:28:13.060 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-05-08T16:28:13.060 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-05-08T16:28:13.060 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-05-08T16:28:13.060 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/test-erasure-eio/x 2024-05-08T16:28:13.060 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-05-08T16:28:14.043 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-05-08T16:28:14.043 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-08T16:28:14.043 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-08T16:28:14.044 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-08T16:28:14.044 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T16:28:14.044 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81110 2024-05-08T16:28:14.044 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.81110/$cluster-$name.asok' 2024-05-08T16:28:14.045 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-05-08T16:28:14.046 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-eio/x '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.81110/$cluster-$name.asok' --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-05-08T16:28:14.080 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:36: run: create_pool rbd 4 2024-05-08T16:28:14.080 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-05-08T16:28:14.422 INFO:tasks.workunit.client.0.smithi172.stderr:pool 'rbd' already exists 2024-05-08T16:28:14.439 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-05-08T16:28:15.442 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: get_asok_path mon.a 2024-05-08T16:28:15.442 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-05-08T16:28:15.442 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-05-08T16:28:15.442 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-08T16:28:15.442 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T16:28:15.443 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81110 2024-05-08T16:28:15.443 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.81110/ceph-mon.a.asok 2024-05-08T16:28:15.443 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: CEPH_ARGS= 2024-05-08T16:28:15.443 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: ceph --admin-daemon /tmp/ceph-asok.81110/ceph-mon.a.asok log flush 2024-05-08T16:28:15.541 INFO:tasks.workunit.client.0.smithi172.stdout:{} 2024-05-08T16:28:15.553 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:40: run: grep 'load: jerasure.*lrc' td/test-erasure-eio/mon.a.log 2024-05-08T16:28:15.556 INFO:tasks.workunit.client.0.smithi172.stdout:2024-05-08T16:28:12.646+0000 7fee09254c80 0 load: jerasure load: lrc 2024-05-08T16:28:15.557 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:41: run: TEST_ec_recovery_multiple_errors td/test-erasure-eio 2024-05-08T16:28:15.557 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:418: TEST_ec_recovery_multiple_errors: local dir=td/test-erasure-eio 2024-05-08T16:28:15.557 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:419: TEST_ec_recovery_multiple_errors: local objname=myobject 2024-05-08T16:28:15.557 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:421: TEST_ec_recovery_multiple_errors: setup_osds 9 2024-05-08T16:28:15.557 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:47: setup_osds: local count=9 2024-05-08T16:28:15.557 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:48: setup_osds: shift 2024-05-08T16:28:15.558 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: expr 9 - 1 2024-05-08T16:28:15.559 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: seq 0 8 2024-05-08T16:28:15.562 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-05-08T16:28:15.563 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 0 2024-05-08T16:28:15.563 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-05-08T16:28:15.563 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-05-08T16:28:15.563 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-05-08T16:28:15.563 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-05-08T16:28:15.563 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/0 2024-05-08T16:28:15.563 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=4457b0f1-028b-482b-8ee3-7ee05861e6cc --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-05-08T16:28:15.563 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-08T16:28:15.563 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-05-08T16:28:15.563 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-08T16:28:15.563 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/0' 2024-05-08T16:28:15.563 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/0/journal' 2024-05-08T16:28:15.563 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-05-08T16:28:15.563 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-05-08T16:28:15.564 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-05-08T16:28:15.564 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-05-08T16:28:15.564 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-08T16:28:15.564 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-08T16:28:15.564 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-08T16:28:15.564 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T16:28:15.564 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81110 2024-05-08T16:28:15.564 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.81110/$cluster-$name.asok' 2024-05-08T16:28:15.564 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.81110/$cluster-$name.asok' 2024-05-08T16:28:15.564 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-05-08T16:28:15.564 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-05-08T16:28:15.564 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-05-08T16:28:15.564 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-05-08T16:28:15.564 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-05-08T16:28:15.565 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-08T16:28:15.565 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-08T16:28:15.565 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-08T16:28:15.565 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-08T16:28:15.565 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-05-08T16:28:15.565 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-05-08T16:28:15.565 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/0 2024-05-08T16:28:15.565 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-05-08T16:28:15.566 INFO:tasks.workunit.client.0.smithi172.stdout:add osd0 336e0b84-8d81-4553-bc0c-9e72002fb23f 2024-05-08T16:28:15.566 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=336e0b84-8d81-4553-bc0c-9e72002fb23f 2024-05-08T16:28:15.566 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 336e0b84-8d81-4553-bc0c-9e72002fb23f' 2024-05-08T16:28:15.566 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-05-08T16:28:15.587 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAfqDtmrstWIxAANNA8PV8tBZeVQEAdcIDkXw== 2024-05-08T16:28:15.587 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAfqDtmrstWIxAANNA8PV8tBZeVQEAdcIDkXw=="}' 2024-05-08T16:28:15.587 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 336e0b84-8d81-4553-bc0c-9e72002fb23f -i td/test-erasure-eio/0/new.json 2024-05-08T16:28:15.915 INFO:tasks.workunit.client.0.smithi172.stdout:0 2024-05-08T16:28:15.931 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/0/new.json 2024-05-08T16:28:15.932 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=4457b0f1-028b-482b-8ee3-7ee05861e6cc --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81110/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAfqDtmrstWIxAANNA8PV8tBZeVQEAdcIDkXw== --osd-uuid 336e0b84-8d81-4553-bc0c-9e72002fb23f 2024-05-08T16:28:15.970 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:28:15.974+0000 7f2bb4dd47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:28:15.992 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:28:15.998+0000 7f2bb4dd47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:28:15.993 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:28:15.998+0000 7f2bb4dd47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:28:15.994 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:28:15.998+0000 7f2bb4dd47c0 -1 bluestore(td/test-erasure-eio/0/block) _read_bdev_label failed to open td/test-erasure-eio/0/block: (2) No such file or directory 2024-05-08T16:28:15.994 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:28:15.998+0000 7f2bb4dd47c0 -1 bluestore(td/test-erasure-eio/0/block) _read_bdev_label failed to open td/test-erasure-eio/0/block: (2) No such file or directory 2024-05-08T16:28:15.994 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:28:15.998+0000 7f2bb4dd47c0 -1 bluestore(td/test-erasure-eio/0/block) _read_bdev_label failed to open td/test-erasure-eio/0/block: (2) No such file or directory 2024-05-08T16:28:16.031 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:28:16.034+0000 7f2bb4dd47c0 -1 bluestore(td/test-erasure-eio/0) _read_fsid unparsable uuid 2024-05-08T16:28:23.237 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/0/keyring 2024-05-08T16:28:23.237 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-05-08T16:28:23.239 INFO:tasks.workunit.client.0.smithi172.stdout:adding osd0 key to auth repository 2024-05-08T16:28:23.239 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-05-08T16:28:23.239 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-05-08T16:28:23.807 INFO:tasks.workunit.client.0.smithi172.stdout:start osd.0 2024-05-08T16:28:23.808 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-05-08T16:28:23.808 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=4457b0f1-028b-482b-8ee3-7ee05861e6cc --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81110/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-05-08T16:28:23.809 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-05-08T16:28:23.809 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-05-08T16:28:23.809 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-05-08T16:28:23.839 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:28:23.842+0000 7f68a3cdf7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:28:23.841 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:28:23.846+0000 7f68a3cdf7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:28:23.842 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:28:23.846+0000 7f68a3cdf7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:28:24.210 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-05-08T16:28:24.211 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-08T16:28:24.211 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-05-08T16:28:24.211 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-08T16:28:24.211 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-08T16:28:24.211 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:28:24.211 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-08T16:28:24.211 INFO:tasks.workunit.client.0.smithi172.stdout:0 2024-05-08T16:28:24.212 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:28:24.212 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:28:24.409 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:28:24.414+0000 7f68a3cdf7c0 -1 Falling back to public interface 2024-05-08T16:28:24.634 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:28:25.635 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:28:25.635 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:28:25.635 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-08T16:28:25.636 INFO:tasks.workunit.client.0.smithi172.stdout:1 2024-05-08T16:28:25.636 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:28:25.636 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:28:26.037 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:28:27.038 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:28:27.039 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:28:27.039 INFO:tasks.workunit.client.0.smithi172.stdout:2 2024-05-08T16:28:27.039 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-08T16:28:27.039 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:28:27.039 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:28:27.068 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:28:27.074+0000 7f68a3cdf7c0 -1 osd.0 0 log_to_monitors true 2024-05-08T16:28:27.633 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:28:28.635 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:28:28.635 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:28:28.636 INFO:tasks.workunit.client.0.smithi172.stdout:3 2024-05-08T16:28:28.636 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-05-08T16:28:28.637 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:28:28.637 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:28:29.164 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:28:30.165 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:28:30.166 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:28:30.166 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-05-08T16:28:30.166 INFO:tasks.workunit.client.0.smithi172.stdout:4 2024-05-08T16:28:30.166 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:28:30.167 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:28:30.615 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:28:31.617 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:28:31.617 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:28:31.617 INFO:tasks.workunit.client.0.smithi172.stdout:5 2024-05-08T16:28:31.618 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-05-08T16:28:31.618 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:28:31.618 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:28:32.095 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:28:33.096 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:28:33.097 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:28:33.097 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-05-08T16:28:33.097 INFO:tasks.workunit.client.0.smithi172.stdout:6 2024-05-08T16:28:33.097 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:28:33.098 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:28:33.549 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:28:34.551 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:28:34.551 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:28:34.551 INFO:tasks.workunit.client.0.smithi172.stdout:7 2024-05-08T16:28:34.552 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-05-08T16:28:34.552 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:28:34.552 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:28:35.103 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:28:36.105 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:28:36.105 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:28:36.105 INFO:tasks.workunit.client.0.smithi172.stdout:8 2024-05-08T16:28:36.105 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-05-08T16:28:36.106 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:28:36.106 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:28:36.563 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:28:37.565 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:28:37.565 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:28:37.565 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-05-08T16:28:37.566 INFO:tasks.workunit.client.0.smithi172.stdout:9 2024-05-08T16:28:37.566 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:28:37.566 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:28:38.086 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:28:39.088 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:28:39.088 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:28:39.088 INFO:tasks.workunit.client.0.smithi172.stdout:10 2024-05-08T16:28:39.089 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-05-08T16:28:39.089 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:28:39.089 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:28:39.614 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:28:40.616 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:28:40.616 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:28:40.616 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-05-08T16:28:40.616 INFO:tasks.workunit.client.0.smithi172.stdout:11 2024-05-08T16:28:40.617 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:28:40.617 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:28:41.242 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:28:42.244 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:28:42.244 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:28:42.245 INFO:tasks.workunit.client.0.smithi172.stdout:12 2024-05-08T16:28:42.245 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-05-08T16:28:42.245 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:28:42.245 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:28:42.908 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:28:43.909 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:28:43.909 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:28:43.910 INFO:tasks.workunit.client.0.smithi172.stdout:13 2024-05-08T16:28:43.910 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-05-08T16:28:43.910 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:28:43.910 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:28:45.058 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:28:46.060 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:28:46.060 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:28:46.060 INFO:tasks.workunit.client.0.smithi172.stdout:14 2024-05-08T16:28:46.061 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-05-08T16:28:46.061 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:28:46.061 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:28:46.902 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:28:47.904 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:28:47.904 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:28:47.904 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-05-08T16:28:47.905 INFO:tasks.workunit.client.0.smithi172.stdout:15 2024-05-08T16:28:47.905 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:28:47.905 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:28:48.818 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:28:49.819 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:28:49.819 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:28:49.820 INFO:tasks.workunit.client.0.smithi172.stdout:16 2024-05-08T16:28:49.820 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-05-08T16:28:49.820 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:28:49.820 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:28:50.712 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:28:51.714 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:28:51.714 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:28:51.714 INFO:tasks.workunit.client.0.smithi172.stdout:17 2024-05-08T16:28:51.715 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-05-08T16:28:51.715 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:28:51.715 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:28:52.390 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:28:53.392 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:28:53.392 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:28:53.392 INFO:tasks.workunit.client.0.smithi172.stdout:18 2024-05-08T16:28:53.393 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-05-08T16:28:53.393 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:28:53.393 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:28:53.832 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:28:54.833 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:28:54.834 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:28:54.834 INFO:tasks.workunit.client.0.smithi172.stdout:19 2024-05-08T16:28:54.834 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-05-08T16:28:54.834 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:28:54.835 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:28:55.270 INFO:tasks.workunit.client.0.smithi172.stdout:osd.0 up in weight 1 up_from 6 up_thru 6 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3814468070,v1:127.0.0.1:6803/3814468070] [v2:127.0.0.1:6804/3814468070,v1:127.0.0.1:6805/3814468070] exists,up 336e0b84-8d81-4553-bc0c-9e72002fb23f 2024-05-08T16:28:55.273 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-08T16:28:55.273 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-08T16:28:55.273 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-08T16:28:55.273 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-05-08T16:28:55.273 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 1 2024-05-08T16:28:55.273 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-05-08T16:28:55.273 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-05-08T16:28:55.273 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-05-08T16:28:55.273 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-05-08T16:28:55.273 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/1 2024-05-08T16:28:55.274 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=4457b0f1-028b-482b-8ee3-7ee05861e6cc --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-05-08T16:28:55.274 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-08T16:28:55.274 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-05-08T16:28:55.274 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-08T16:28:55.274 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/1' 2024-05-08T16:28:55.274 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/1/journal' 2024-05-08T16:28:55.274 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-05-08T16:28:55.274 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-05-08T16:28:55.274 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-05-08T16:28:55.274 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-05-08T16:28:55.274 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-08T16:28:55.274 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-08T16:28:55.274 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-08T16:28:55.275 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T16:28:55.275 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81110 2024-05-08T16:28:55.275 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.81110/$cluster-$name.asok' 2024-05-08T16:28:55.275 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.81110/$cluster-$name.asok' 2024-05-08T16:28:55.275 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-05-08T16:28:55.275 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-05-08T16:28:55.275 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-05-08T16:28:55.275 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-05-08T16:28:55.275 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-05-08T16:28:55.275 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-08T16:28:55.275 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-08T16:28:55.275 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-08T16:28:55.275 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-08T16:28:55.275 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-05-08T16:28:55.276 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-05-08T16:28:55.276 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/1 2024-05-08T16:28:55.277 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-05-08T16:28:55.277 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=1a306cc0-7528-4519-8f66-fc5ac46a9987 2024-05-08T16:28:55.277 INFO:tasks.workunit.client.0.smithi172.stdout:add osd1 1a306cc0-7528-4519-8f66-fc5ac46a9987 2024-05-08T16:28:55.278 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 1a306cc0-7528-4519-8f66-fc5ac46a9987' 2024-05-08T16:28:55.278 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-05-08T16:28:55.298 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBHqDtmVRsiEhAAv1nMPK/Huwli6RbKuOZnZQ== 2024-05-08T16:28:55.298 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBHqDtmVRsiEhAAv1nMPK/Huwli6RbKuOZnZQ=="}' 2024-05-08T16:28:55.298 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 1a306cc0-7528-4519-8f66-fc5ac46a9987 -i td/test-erasure-eio/1/new.json 2024-05-08T16:28:55.793 INFO:tasks.workunit.client.0.smithi172.stdout:1 2024-05-08T16:28:55.810 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/1/new.json 2024-05-08T16:28:55.811 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=4457b0f1-028b-482b-8ee3-7ee05861e6cc --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81110/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBHqDtmVRsiEhAAv1nMPK/Huwli6RbKuOZnZQ== --osd-uuid 1a306cc0-7528-4519-8f66-fc5ac46a9987 2024-05-08T16:28:55.845 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:28:55.849+0000 7f577443d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:28:55.878 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:28:55.881+0000 7f577443d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:28:55.880 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:28:55.885+0000 7f577443d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:28:55.880 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:28:55.885+0000 7f577443d7c0 -1 bluestore(td/test-erasure-eio/1/block) _read_bdev_label failed to open td/test-erasure-eio/1/block: (2) No such file or directory 2024-05-08T16:28:55.880 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:28:55.885+0000 7f577443d7c0 -1 bluestore(td/test-erasure-eio/1/block) _read_bdev_label failed to open td/test-erasure-eio/1/block: (2) No such file or directory 2024-05-08T16:28:55.880 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:28:55.885+0000 7f577443d7c0 -1 bluestore(td/test-erasure-eio/1/block) _read_bdev_label failed to open td/test-erasure-eio/1/block: (2) No such file or directory 2024-05-08T16:28:55.925 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:28:55.929+0000 7f577443d7c0 -1 bluestore(td/test-erasure-eio/1) _read_fsid unparsable uuid 2024-05-08T16:29:02.729 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/1/keyring 2024-05-08T16:29:02.729 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-05-08T16:29:02.730 INFO:tasks.workunit.client.0.smithi172.stdout:adding osd1 key to auth repository 2024-05-08T16:29:02.730 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-05-08T16:29:02.730 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-05-08T16:29:03.279 INFO:tasks.workunit.client.0.smithi172.stdout:start osd.1 2024-05-08T16:29:03.279 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-05-08T16:29:03.279 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=4457b0f1-028b-482b-8ee3-7ee05861e6cc --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81110/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-05-08T16:29:03.280 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-05-08T16:29:03.280 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-05-08T16:29:03.280 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-05-08T16:29:03.310 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:29:03.313+0000 7f1c668e47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:29:03.312 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:29:03.317+0000 7f1c668e47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:29:03.313 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:29:03.317+0000 7f1c668e47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:29:03.690 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-05-08T16:29:03.690 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-08T16:29:03.691 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-05-08T16:29:03.691 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-08T16:29:03.691 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-08T16:29:03.691 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:29:03.691 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-08T16:29:03.691 INFO:tasks.workunit.client.0.smithi172.stdout:0 2024-05-08T16:29:03.691 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:29:03.692 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T16:29:03.907 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:29:03.909+0000 7f1c668e47c0 -1 Falling back to public interface 2024-05-08T16:29:04.119 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:29:05.121 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:29:05.121 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:29:05.121 INFO:tasks.workunit.client.0.smithi172.stdout:1 2024-05-08T16:29:05.122 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-08T16:29:05.122 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:29:05.122 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T16:29:05.542 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:29:06.544 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:29:06.544 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:29:06.544 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-08T16:29:06.545 INFO:tasks.workunit.client.0.smithi172.stdout:2 2024-05-08T16:29:06.545 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:29:06.549+0000 7f1c668e47c0 -1 osd.1 0 log_to_monitors true 2024-05-08T16:29:06.545 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:29:06.545 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T16:29:06.946 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:29:07.948 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:29:07.948 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:29:07.948 INFO:tasks.workunit.client.0.smithi172.stdout:3 2024-05-08T16:29:07.949 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-05-08T16:29:07.949 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:29:07.949 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T16:29:08.399 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:29:09.400 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:29:09.401 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:29:09.401 INFO:tasks.workunit.client.0.smithi172.stdout:4 2024-05-08T16:29:09.401 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-05-08T16:29:09.402 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:29:09.402 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T16:29:09.924 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:29:10.926 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:29:10.926 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:29:10.926 INFO:tasks.workunit.client.0.smithi172.stdout:5 2024-05-08T16:29:10.927 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-05-08T16:29:10.927 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:29:10.927 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T16:29:11.498 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:29:12.499 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:29:12.499 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:29:12.500 INFO:tasks.workunit.client.0.smithi172.stdout:6 2024-05-08T16:29:12.500 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-05-08T16:29:12.500 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:29:12.500 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T16:29:12.976 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:29:13.977 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:29:13.978 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:29:13.978 INFO:tasks.workunit.client.0.smithi172.stdout:7 2024-05-08T16:29:13.978 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-05-08T16:29:13.979 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:29:13.979 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T16:29:14.547 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:29:15.549 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:29:15.549 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:29:15.549 INFO:tasks.workunit.client.0.smithi172.stdout:8 2024-05-08T16:29:15.550 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-05-08T16:29:15.550 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:29:15.550 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T16:29:16.014 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:29:17.016 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:29:17.016 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:29:17.016 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-05-08T16:29:17.017 INFO:tasks.workunit.client.0.smithi172.stdout:9 2024-05-08T16:29:17.017 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:29:17.017 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T16:29:17.554 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:29:18.556 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:29:18.556 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:29:18.557 INFO:tasks.workunit.client.0.smithi172.stdout:10 2024-05-08T16:29:18.557 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-05-08T16:29:18.557 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:29:18.557 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T16:29:19.024 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:29:20.025 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:29:20.025 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:29:20.025 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-05-08T16:29:20.026 INFO:tasks.workunit.client.0.smithi172.stdout:11 2024-05-08T16:29:20.026 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:29:20.026 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T16:29:20.535 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:29:21.536 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:29:21.537 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:29:21.537 INFO:tasks.workunit.client.0.smithi172.stdout:12 2024-05-08T16:29:21.538 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-05-08T16:29:21.538 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:29:21.538 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T16:29:22.203 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:29:23.205 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:29:23.205 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:29:23.206 INFO:tasks.workunit.client.0.smithi172.stdout:13 2024-05-08T16:29:23.206 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-05-08T16:29:23.206 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:29:23.207 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T16:29:24.011 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:29:25.013 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:29:25.013 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:29:25.013 INFO:tasks.workunit.client.0.smithi172.stdout:14 2024-05-08T16:29:25.014 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-05-08T16:29:25.014 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:29:25.014 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T16:29:26.094 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:29:27.095 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:29:27.095 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:29:27.095 INFO:tasks.workunit.client.0.smithi172.stdout:15 2024-05-08T16:29:27.096 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-05-08T16:29:27.096 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:29:27.096 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T16:29:28.005 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:29:29.007 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:29:29.007 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:29:29.007 INFO:tasks.workunit.client.0.smithi172.stdout:16 2024-05-08T16:29:29.008 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-05-08T16:29:29.008 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:29:29.008 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T16:29:29.864 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:29:30.866 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:29:30.866 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:29:30.866 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-05-08T16:29:30.866 INFO:tasks.workunit.client.0.smithi172.stdout:17 2024-05-08T16:29:30.867 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:29:30.867 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T16:29:31.480 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:29:32.483 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:29:32.483 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:29:32.483 INFO:tasks.workunit.client.0.smithi172.stdout:18 2024-05-08T16:29:32.484 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-05-08T16:29:32.484 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:29:32.484 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T16:29:33.456 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:29:34.458 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:29:34.458 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:29:34.459 INFO:tasks.workunit.client.0.smithi172.stdout:19 2024-05-08T16:29:34.459 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-05-08T16:29:34.459 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:29:34.459 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T16:29:35.051 INFO:tasks.workunit.client.0.smithi172.stdout:osd.1 up in weight 1 up_from 13 up_thru 13 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/280190551,v1:127.0.0.1:6811/280190551] [v2:127.0.0.1:6812/280190551,v1:127.0.0.1:6813/280190551] exists,up 1a306cc0-7528-4519-8f66-fc5ac46a9987 2024-05-08T16:29:35.055 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-08T16:29:35.055 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-08T16:29:35.055 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-08T16:29:35.055 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-05-08T16:29:35.055 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 2 2024-05-08T16:29:35.055 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-05-08T16:29:35.055 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-05-08T16:29:35.055 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-05-08T16:29:35.055 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-05-08T16:29:35.055 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/2 2024-05-08T16:29:35.055 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=4457b0f1-028b-482b-8ee3-7ee05861e6cc --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-05-08T16:29:35.055 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-08T16:29:35.056 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-05-08T16:29:35.056 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-08T16:29:35.056 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/2' 2024-05-08T16:29:35.056 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/2/journal' 2024-05-08T16:29:35.056 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-05-08T16:29:35.056 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-05-08T16:29:35.056 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-05-08T16:29:35.056 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-05-08T16:29:35.056 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-08T16:29:35.056 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-08T16:29:35.056 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-08T16:29:35.056 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T16:29:35.056 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81110 2024-05-08T16:29:35.056 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.81110/$cluster-$name.asok' 2024-05-08T16:29:35.057 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.81110/$cluster-$name.asok' 2024-05-08T16:29:35.057 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-05-08T16:29:35.057 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-05-08T16:29:35.057 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-05-08T16:29:35.057 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-05-08T16:29:35.057 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-05-08T16:29:35.057 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-08T16:29:35.057 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-08T16:29:35.057 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-08T16:29:35.057 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-08T16:29:35.057 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-05-08T16:29:35.057 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-05-08T16:29:35.057 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/2 2024-05-08T16:29:35.058 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-05-08T16:29:35.058 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=7fa184fe-823a-4b6d-a6ab-451dd0d823a6 2024-05-08T16:29:35.058 INFO:tasks.workunit.client.0.smithi172.stdout:add osd2 7fa184fe-823a-4b6d-a6ab-451dd0d823a6 2024-05-08T16:29:35.059 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 7fa184fe-823a-4b6d-a6ab-451dd0d823a6' 2024-05-08T16:29:35.059 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-05-08T16:29:35.079 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBvqDtmLF8bBRAAxsCtSq2SepIYGbE/BKDPzg== 2024-05-08T16:29:35.080 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBvqDtmLF8bBRAAxsCtSq2SepIYGbE/BKDPzg=="}' 2024-05-08T16:29:35.080 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 7fa184fe-823a-4b6d-a6ab-451dd0d823a6 -i td/test-erasure-eio/2/new.json 2024-05-08T16:29:35.609 INFO:tasks.workunit.client.0.smithi172.stdout:2 2024-05-08T16:29:35.626 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/2/new.json 2024-05-08T16:29:35.627 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=4457b0f1-028b-482b-8ee3-7ee05861e6cc --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81110/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBvqDtmLF8bBRAAxsCtSq2SepIYGbE/BKDPzg== --osd-uuid 7fa184fe-823a-4b6d-a6ab-451dd0d823a6 2024-05-08T16:29:35.656 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:29:35.661+0000 7f3c92d857c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:29:35.658 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:29:35.661+0000 7f3c92d857c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:29:35.660 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:29:35.665+0000 7f3c92d857c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:29:35.660 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:29:35.665+0000 7f3c92d857c0 -1 bluestore(td/test-erasure-eio/2/block) _read_bdev_label failed to open td/test-erasure-eio/2/block: (2) No such file or directory 2024-05-08T16:29:35.660 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:29:35.665+0000 7f3c92d857c0 -1 bluestore(td/test-erasure-eio/2/block) _read_bdev_label failed to open td/test-erasure-eio/2/block: (2) No such file or directory 2024-05-08T16:29:35.660 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:29:35.665+0000 7f3c92d857c0 -1 bluestore(td/test-erasure-eio/2/block) _read_bdev_label failed to open td/test-erasure-eio/2/block: (2) No such file or directory 2024-05-08T16:29:35.733 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:29:35.737+0000 7f3c92d857c0 -1 bluestore(td/test-erasure-eio/2) _read_fsid unparsable uuid 2024-05-08T16:29:42.954 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/2/keyring 2024-05-08T16:29:42.954 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-05-08T16:29:42.955 INFO:tasks.workunit.client.0.smithi172.stdout:adding osd2 key to auth repository 2024-05-08T16:29:42.955 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-05-08T16:29:42.955 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-05-08T16:29:43.517 INFO:tasks.workunit.client.0.smithi172.stdout:start osd.2 2024-05-08T16:29:43.518 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-05-08T16:29:43.518 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=4457b0f1-028b-482b-8ee3-7ee05861e6cc --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81110/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-05-08T16:29:43.518 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-05-08T16:29:43.518 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-05-08T16:29:43.518 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-05-08T16:29:43.547 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:29:43.549+0000 7f3c9f6ff7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:29:43.549 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:29:43.553+0000 7f3c9f6ff7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:29:43.551 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:29:43.553+0000 7f3c9f6ff7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:29:43.912 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-05-08T16:29:43.912 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-08T16:29:43.913 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-05-08T16:29:43.913 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-08T16:29:43.913 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-08T16:29:43.913 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:29:43.913 INFO:tasks.workunit.client.0.smithi172.stdout:0 2024-05-08T16:29:43.913 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-08T16:29:43.914 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:29:43.914 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T16:29:44.110 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:29:44.112+0000 7f3c9f6ff7c0 -1 Falling back to public interface 2024-05-08T16:29:44.374 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:29:45.375 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:29:45.376 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:29:45.376 INFO:tasks.workunit.client.0.smithi172.stdout:1 2024-05-08T16:29:45.376 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-08T16:29:45.376 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:29:45.376 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T16:29:45.793 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:29:46.795 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:29:46.795 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:29:46.796 INFO:tasks.workunit.client.0.smithi172.stdout:2 2024-05-08T16:29:46.796 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-08T16:29:46.796 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:29:46.796 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T16:29:46.872 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:29:46.876+0000 7f3c9f6ff7c0 -1 osd.2 0 log_to_monitors true 2024-05-08T16:29:47.281 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:29:48.283 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:29:48.284 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:29:48.284 INFO:tasks.workunit.client.0.smithi172.stdout:3 2024-05-08T16:29:48.284 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-05-08T16:29:48.284 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:29:48.285 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T16:29:48.762 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:29:49.764 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:29:49.764 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:29:49.764 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-05-08T16:29:49.764 INFO:tasks.workunit.client.0.smithi172.stdout:4 2024-05-08T16:29:49.765 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:29:49.765 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T16:29:50.261 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:29:51.263 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:29:51.264 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:29:51.264 INFO:tasks.workunit.client.0.smithi172.stdout:5 2024-05-08T16:29:51.265 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-05-08T16:29:51.265 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:29:51.265 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T16:29:51.834 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:29:52.836 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:29:52.836 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:29:52.836 INFO:tasks.workunit.client.0.smithi172.stdout:6 2024-05-08T16:29:52.836 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-05-08T16:29:52.837 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:29:52.837 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T16:29:53.450 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:29:54.452 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:29:54.452 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:29:54.453 INFO:tasks.workunit.client.0.smithi172.stdout:7 2024-05-08T16:29:54.453 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-05-08T16:29:54.453 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:29:54.453 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T16:29:55.079 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:29:56.081 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:29:56.081 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:29:56.081 INFO:tasks.workunit.client.0.smithi172.stdout:8 2024-05-08T16:29:56.082 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-05-08T16:29:56.082 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:29:56.082 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T16:29:56.549 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:29:57.551 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:29:57.551 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:29:57.551 INFO:tasks.workunit.client.0.smithi172.stdout:9 2024-05-08T16:29:57.552 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-05-08T16:29:57.552 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:29:57.552 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T16:29:58.051 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:29:59.052 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:29:59.052 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:29:59.053 INFO:tasks.workunit.client.0.smithi172.stdout:10 2024-05-08T16:29:59.053 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-05-08T16:29:59.053 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:29:59.053 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T16:29:59.606 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:30:00.608 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:30:00.608 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:30:00.608 INFO:tasks.workunit.client.0.smithi172.stdout:11 2024-05-08T16:30:00.609 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-05-08T16:30:00.609 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:30:00.609 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T16:30:01.236 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:30:02.238 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:30:02.238 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:30:02.239 INFO:tasks.workunit.client.0.smithi172.stdout:12 2024-05-08T16:30:02.239 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-05-08T16:30:02.239 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:30:02.239 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T16:30:02.645 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:30:03.647 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:30:03.647 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:30:03.647 INFO:tasks.workunit.client.0.smithi172.stdout:13 2024-05-08T16:30:03.648 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-05-08T16:30:03.648 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:30:03.648 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T16:30:04.424 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:30:05.426 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:30:05.426 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:30:05.426 INFO:tasks.workunit.client.0.smithi172.stdout:14 2024-05-08T16:30:05.427 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-05-08T16:30:05.427 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:30:05.427 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T16:30:06.623 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:30:07.625 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:30:07.625 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:30:07.625 INFO:tasks.workunit.client.0.smithi172.stdout:15 2024-05-08T16:30:07.626 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-05-08T16:30:07.626 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:30:07.626 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T16:30:08.261 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:30:09.262 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:30:09.263 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:30:09.263 INFO:tasks.workunit.client.0.smithi172.stdout:16 2024-05-08T16:30:09.263 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-05-08T16:30:09.263 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:30:09.264 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T16:30:10.094 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:30:11.096 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:30:11.096 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:30:11.096 INFO:tasks.workunit.client.0.smithi172.stdout:17 2024-05-08T16:30:11.097 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-05-08T16:30:11.097 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:30:11.097 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T16:30:12.223 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:30:13.225 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:30:13.225 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:30:13.226 INFO:tasks.workunit.client.0.smithi172.stdout:18 2024-05-08T16:30:13.226 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-05-08T16:30:13.227 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:30:13.227 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T16:30:13.915 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:30:14.917 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:30:14.917 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:30:14.917 INFO:tasks.workunit.client.0.smithi172.stdout:19 2024-05-08T16:30:14.918 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-05-08T16:30:14.918 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:30:14.918 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T16:30:15.549 INFO:tasks.workunit.client.0.smithi172.stdout:osd.2 up in weight 1 up_from 19 up_thru 19 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/3688366951,v1:127.0.0.1:6819/3688366951] [v2:127.0.0.1:6820/3688366951,v1:127.0.0.1:6821/3688366951] exists,up 7fa184fe-823a-4b6d-a6ab-451dd0d823a6 2024-05-08T16:30:15.552 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-08T16:30:15.552 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-08T16:30:15.552 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-08T16:30:15.552 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-05-08T16:30:15.552 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 3 2024-05-08T16:30:15.552 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-05-08T16:30:15.552 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-05-08T16:30:15.552 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2024-05-08T16:30:15.552 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-05-08T16:30:15.553 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/3 2024-05-08T16:30:15.553 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=4457b0f1-028b-482b-8ee3-7ee05861e6cc --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-05-08T16:30:15.553 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-08T16:30:15.553 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-05-08T16:30:15.553 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-08T16:30:15.553 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/3' 2024-05-08T16:30:15.553 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/3/journal' 2024-05-08T16:30:15.553 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-05-08T16:30:15.553 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-05-08T16:30:15.553 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-05-08T16:30:15.553 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-05-08T16:30:15.553 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-08T16:30:15.553 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-08T16:30:15.553 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-08T16:30:15.553 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T16:30:15.554 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81110 2024-05-08T16:30:15.554 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.81110/$cluster-$name.asok' 2024-05-08T16:30:15.556 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.81110/$cluster-$name.asok' 2024-05-08T16:30:15.556 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-05-08T16:30:15.556 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-05-08T16:30:15.556 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-05-08T16:30:15.556 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-05-08T16:30:15.556 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-05-08T16:30:15.556 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-08T16:30:15.556 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-08T16:30:15.556 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-08T16:30:15.556 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-08T16:30:15.556 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-05-08T16:30:15.557 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-05-08T16:30:15.557 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/3 2024-05-08T16:30:15.557 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-05-08T16:30:15.557 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=7085f7ad-c597-4159-9afe-f81f72077a14 2024-05-08T16:30:15.557 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 7085f7ad-c597-4159-9afe-f81f72077a14' 2024-05-08T16:30:15.557 INFO:tasks.workunit.client.0.smithi172.stdout:add osd3 7085f7ad-c597-4159-9afe-f81f72077a14 2024-05-08T16:30:15.557 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-05-08T16:30:15.580 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCXqDtmGy3qIhAAfBijxjQuAxgAN3a5BJJaKg== 2024-05-08T16:30:15.580 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCXqDtmGy3qIhAAfBijxjQuAxgAN3a5BJJaKg=="}' 2024-05-08T16:30:15.580 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 7085f7ad-c597-4159-9afe-f81f72077a14 -i td/test-erasure-eio/3/new.json 2024-05-08T16:30:17.074 INFO:tasks.workunit.client.0.smithi172.stdout:3 2024-05-08T16:30:17.096 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/3/new.json 2024-05-08T16:30:17.098 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=4457b0f1-028b-482b-8ee3-7ee05861e6cc --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81110/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCXqDtmGy3qIhAAfBijxjQuAxgAN3a5BJJaKg== --osd-uuid 7085f7ad-c597-4159-9afe-f81f72077a14 2024-05-08T16:30:17.134 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:30:17.136+0000 7f07099097c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:30:17.181 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:30:17.184+0000 7f07099097c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:30:17.182 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:30:17.184+0000 7f07099097c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:30:17.183 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:30:17.184+0000 7f07099097c0 -1 bluestore(td/test-erasure-eio/3/block) _read_bdev_label failed to open td/test-erasure-eio/3/block: (2) No such file or directory 2024-05-08T16:30:17.183 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:30:17.184+0000 7f07099097c0 -1 bluestore(td/test-erasure-eio/3/block) _read_bdev_label failed to open td/test-erasure-eio/3/block: (2) No such file or directory 2024-05-08T16:30:17.184 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:30:17.184+0000 7f07099097c0 -1 bluestore(td/test-erasure-eio/3/block) _read_bdev_label failed to open td/test-erasure-eio/3/block: (2) No such file or directory 2024-05-08T16:30:17.322 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:30:17.324+0000 7f07099097c0 -1 bluestore(td/test-erasure-eio/3) _read_fsid unparsable uuid 2024-05-08T16:30:24.473 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/3/keyring 2024-05-08T16:30:24.473 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-05-08T16:30:24.474 INFO:tasks.workunit.client.0.smithi172.stdout:adding osd3 key to auth repository 2024-05-08T16:30:24.474 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2024-05-08T16:30:24.474 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-05-08T16:30:25.081 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2024-05-08T16:30:25.082 INFO:tasks.workunit.client.0.smithi172.stdout:start osd.3 2024-05-08T16:30:25.082 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-05-08T16:30:25.083 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=4457b0f1-028b-482b-8ee3-7ee05861e6cc --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81110/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-05-08T16:30:25.083 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-05-08T16:30:25.083 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-05-08T16:30:25.113 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:30:25.115+0000 7f23bf5527c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:30:25.115 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:30:25.115+0000 7f23bf5527c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:30:25.117 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:30:25.119+0000 7f23bf5527c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:30:25.525 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2024-05-08T16:30:25.525 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-08T16:30:25.525 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-05-08T16:30:25.525 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-08T16:30:25.525 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-08T16:30:25.525 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:30:25.526 INFO:tasks.workunit.client.0.smithi172.stdout:0 2024-05-08T16:30:25.526 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-08T16:30:25.526 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:30:25.526 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T16:30:25.682 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:30:25.683+0000 7f23bf5527c0 -1 Falling back to public interface 2024-05-08T16:30:25.986 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:30:26.988 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:30:26.988 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:30:26.988 INFO:tasks.workunit.client.0.smithi172.stdout:1 2024-05-08T16:30:26.989 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-08T16:30:26.989 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:30:26.989 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T16:30:27.400 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:30:28.380 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:30:28.383+0000 7f23bf5527c0 -1 osd.3 0 log_to_monitors true 2024-05-08T16:30:28.402 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:30:28.402 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:30:28.402 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-08T16:30:28.403 INFO:tasks.workunit.client.0.smithi172.stdout:2 2024-05-08T16:30:28.403 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T16:30:28.403 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:30:29.248 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:30:30.251 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:30:30.251 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:30:30.251 INFO:tasks.workunit.client.0.smithi172.stdout:3 2024-05-08T16:30:30.251 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-05-08T16:30:30.252 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:30:30.252 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T16:30:31.067 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:30:32.068 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:30:32.069 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:30:32.069 INFO:tasks.workunit.client.0.smithi172.stdout:4 2024-05-08T16:30:32.069 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-05-08T16:30:32.070 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:30:32.070 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T16:30:32.705 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:30:33.707 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:30:33.708 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:30:33.708 INFO:tasks.workunit.client.0.smithi172.stdout:5 2024-05-08T16:30:33.709 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-05-08T16:30:33.709 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:30:33.709 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T16:30:34.365 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:30:35.367 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:30:35.367 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:30:35.367 INFO:tasks.workunit.client.0.smithi172.stdout:6 2024-05-08T16:30:35.368 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-05-08T16:30:35.368 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:30:35.368 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T16:30:35.843 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:30:36.845 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:30:36.845 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:30:36.846 INFO:tasks.workunit.client.0.smithi172.stdout:7 2024-05-08T16:30:36.846 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-05-08T16:30:36.846 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:30:36.846 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T16:30:37.402 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:30:38.403 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:30:38.404 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:30:38.404 INFO:tasks.workunit.client.0.smithi172.stdout:8 2024-05-08T16:30:38.404 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-05-08T16:30:38.405 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:30:38.405 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T16:30:38.877 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:30:39.879 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:30:39.879 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:30:39.880 INFO:tasks.workunit.client.0.smithi172.stdout:9 2024-05-08T16:30:39.880 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-05-08T16:30:39.880 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:30:39.880 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T16:30:40.338 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:30:41.340 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:30:41.341 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:30:41.341 INFO:tasks.workunit.client.0.smithi172.stdout:10 2024-05-08T16:30:41.341 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-05-08T16:30:41.341 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:30:41.341 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T16:30:41.826 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:30:42.827 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:30:42.827 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:30:42.828 INFO:tasks.workunit.client.0.smithi172.stdout:11 2024-05-08T16:30:42.828 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-05-08T16:30:42.828 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:30:42.828 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T16:30:43.339 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:30:44.341 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:30:44.341 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:30:44.341 INFO:tasks.workunit.client.0.smithi172.stdout:12 2024-05-08T16:30:44.342 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-05-08T16:30:44.342 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:30:44.342 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T16:30:44.883 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:30:45.885 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:30:45.885 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:30:45.885 INFO:tasks.workunit.client.0.smithi172.stdout:13 2024-05-08T16:30:45.886 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-05-08T16:30:45.886 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:30:45.886 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T16:30:46.830 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:30:47.831 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:30:47.831 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:30:47.831 INFO:tasks.workunit.client.0.smithi172.stdout:14 2024-05-08T16:30:47.832 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-05-08T16:30:47.832 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:30:47.832 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T16:30:48.825 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:30:49.827 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:30:49.827 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:30:49.828 INFO:tasks.workunit.client.0.smithi172.stdout:15 2024-05-08T16:30:49.828 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-05-08T16:30:49.828 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:30:49.828 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T16:30:50.707 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:30:51.709 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:30:51.709 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:30:51.710 INFO:tasks.workunit.client.0.smithi172.stdout:16 2024-05-08T16:30:51.710 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-05-08T16:30:51.710 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:30:51.710 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T16:30:52.774 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:30:53.776 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:30:53.777 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:30:53.777 INFO:tasks.workunit.client.0.smithi172.stdout:17 2024-05-08T16:30:53.778 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-05-08T16:30:53.778 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:30:53.778 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T16:30:55.065 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:30:56.067 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:30:56.067 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:30:56.067 INFO:tasks.workunit.client.0.smithi172.stdout:18 2024-05-08T16:30:56.068 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-05-08T16:30:56.068 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:30:56.069 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T16:30:56.711 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:30:57.712 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:30:57.713 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:30:57.713 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-05-08T16:30:57.713 INFO:tasks.workunit.client.0.smithi172.stdout:19 2024-05-08T16:30:57.713 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:30:57.713 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T16:30:58.135 INFO:tasks.workunit.client.0.smithi172.stdout:osd.3 up in weight 1 up_from 28 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/3782679226,v1:127.0.0.1:6827/3782679226] [v2:127.0.0.1:6828/3782679226,v1:127.0.0.1:6829/3782679226] exists,up 7085f7ad-c597-4159-9afe-f81f72077a14 2024-05-08T16:30:58.139 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-08T16:30:58.139 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-08T16:30:58.139 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-08T16:30:58.139 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-05-08T16:30:58.139 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 4 2024-05-08T16:30:58.139 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-05-08T16:30:58.139 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-05-08T16:30:58.139 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=4 2024-05-08T16:30:58.139 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-05-08T16:30:58.139 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/4 2024-05-08T16:30:58.139 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=4457b0f1-028b-482b-8ee3-7ee05861e6cc --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-05-08T16:30:58.140 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-08T16:30:58.140 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-05-08T16:30:58.140 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-08T16:30:58.140 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/4' 2024-05-08T16:30:58.140 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/4/journal' 2024-05-08T16:30:58.140 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-05-08T16:30:58.140 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-05-08T16:30:58.140 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-05-08T16:30:58.140 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-05-08T16:30:58.140 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-08T16:30:58.140 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-08T16:30:58.140 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-08T16:30:58.140 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T16:30:58.140 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81110 2024-05-08T16:30:58.141 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.81110/$cluster-$name.asok' 2024-05-08T16:30:58.141 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.81110/$cluster-$name.asok' 2024-05-08T16:30:58.141 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-05-08T16:30:58.141 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-05-08T16:30:58.141 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-05-08T16:30:58.141 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-05-08T16:30:58.141 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-05-08T16:30:58.141 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-08T16:30:58.141 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-08T16:30:58.141 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-08T16:30:58.141 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-08T16:30:58.141 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-05-08T16:30:58.141 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-05-08T16:30:58.141 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/4 2024-05-08T16:30:58.142 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-05-08T16:30:58.142 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=21f94280-e9a5-4de7-88e6-9d4371bc9808 2024-05-08T16:30:58.142 INFO:tasks.workunit.client.0.smithi172.stdout:add osd4 21f94280-e9a5-4de7-88e6-9d4371bc9808 2024-05-08T16:30:58.143 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd4 21f94280-e9a5-4de7-88e6-9d4371bc9808' 2024-05-08T16:30:58.143 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-05-08T16:30:58.162 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDCqDtmZbkOChAAVzjkQ4f8d4xIaJDD0arJOQ== 2024-05-08T16:30:58.162 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDCqDtmZbkOChAAVzjkQ4f8d4xIaJDD0arJOQ=="}' 2024-05-08T16:30:58.163 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 21f94280-e9a5-4de7-88e6-9d4371bc9808 -i td/test-erasure-eio/4/new.json 2024-05-08T16:30:59.034 INFO:tasks.workunit.client.0.smithi172.stdout:4 2024-05-08T16:30:59.053 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/4/new.json 2024-05-08T16:30:59.054 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 4 --fsid=4457b0f1-028b-482b-8ee3-7ee05861e6cc --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/4 --osd-journal=td/test-erasure-eio/4/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81110/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDCqDtmZbkOChAAVzjkQ4f8d4xIaJDD0arJOQ== --osd-uuid 21f94280-e9a5-4de7-88e6-9d4371bc9808 2024-05-08T16:30:59.088 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:30:59.090+0000 7ff186efd7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:30:59.090 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:30:59.094+0000 7ff186efd7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:30:59.091 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:30:59.094+0000 7ff186efd7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:30:59.091 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:30:59.094+0000 7ff186efd7c0 -1 bluestore(td/test-erasure-eio/4/block) _read_bdev_label failed to open td/test-erasure-eio/4/block: (2) No such file or directory 2024-05-08T16:30:59.091 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:30:59.094+0000 7ff186efd7c0 -1 bluestore(td/test-erasure-eio/4/block) _read_bdev_label failed to open td/test-erasure-eio/4/block: (2) No such file or directory 2024-05-08T16:30:59.092 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:30:59.094+0000 7ff186efd7c0 -1 bluestore(td/test-erasure-eio/4/block) _read_bdev_label failed to open td/test-erasure-eio/4/block: (2) No such file or directory 2024-05-08T16:30:59.245 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:30:59.250+0000 7ff186efd7c0 -1 bluestore(td/test-erasure-eio/4) _read_fsid unparsable uuid 2024-05-08T16:31:06.861 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/4/keyring 2024-05-08T16:31:06.862 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-05-08T16:31:06.862 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd4 key to auth repository 2024-05-08T16:31:06.863 INFO:tasks.workunit.client.0.smithi172.stdout:adding osd4 key to auth repository 2024-05-08T16:31:06.863 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-05-08T16:31:07.475 INFO:tasks.workunit.client.0.smithi172.stdout:start osd.4 2024-05-08T16:31:07.475 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.4 2024-05-08T16:31:07.476 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-05-08T16:31:07.476 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-05-08T16:31:07.476 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-05-08T16:31:07.476 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 4 --fsid=4457b0f1-028b-482b-8ee3-7ee05861e6cc --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/4 --osd-journal=td/test-erasure-eio/4/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81110/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-05-08T16:31:07.506 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:31:07.510+0000 7fd712b457c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:31:07.509 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:31:07.514+0000 7fd712b457c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:31:07.510 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:31:07.514+0000 7fd712b457c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:31:07.935 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 4 2024-05-08T16:31:07.935 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-08T16:31:07.935 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=4 2024-05-08T16:31:07.935 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-08T16:31:07.935 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-08T16:31:07.935 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:31:07.935 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-08T16:31:07.936 INFO:tasks.workunit.client.0.smithi172.stdout:0 2024-05-08T16:31:07.936 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:31:07.936 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-08T16:31:08.057 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:31:08.062+0000 7fd712b457c0 -1 Falling back to public interface 2024-05-08T16:31:08.345 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:31:09.346 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:31:09.346 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:31:09.346 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-08T16:31:09.347 INFO:tasks.workunit.client.0.smithi172.stdout:1 2024-05-08T16:31:09.347 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:31:09.347 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-08T16:31:09.750 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:31:10.752 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:31:10.752 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:31:10.753 INFO:tasks.workunit.client.0.smithi172.stdout:2 2024-05-08T16:31:10.753 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-08T16:31:10.753 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:31:10.753 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-08T16:31:10.776 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:31:10.782+0000 7fd712b457c0 -1 osd.4 0 log_to_monitors true 2024-05-08T16:31:11.222 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:31:12.224 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:31:12.224 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:31:12.225 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-05-08T16:31:12.225 INFO:tasks.workunit.client.0.smithi172.stdout:3 2024-05-08T16:31:12.225 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:31:12.225 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-08T16:31:12.695 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:31:13.697 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:31:13.697 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:31:13.697 INFO:tasks.workunit.client.0.smithi172.stdout:4 2024-05-08T16:31:13.698 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-05-08T16:31:13.698 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:31:13.698 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-08T16:31:14.219 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:31:15.221 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:31:15.221 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:31:15.222 INFO:tasks.workunit.client.0.smithi172.stdout:5 2024-05-08T16:31:15.222 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-05-08T16:31:15.222 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:31:15.222 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-08T16:31:15.871 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:31:16.872 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:31:16.872 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:31:16.872 INFO:tasks.workunit.client.0.smithi172.stdout:6 2024-05-08T16:31:16.873 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-05-08T16:31:16.873 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:31:16.873 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-08T16:31:17.531 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:31:18.533 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:31:18.533 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:31:18.533 INFO:tasks.workunit.client.0.smithi172.stdout:7 2024-05-08T16:31:18.534 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-05-08T16:31:18.534 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:31:18.534 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-08T16:31:19.138 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:31:20.140 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:31:20.140 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:31:20.140 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-05-08T16:31:20.141 INFO:tasks.workunit.client.0.smithi172.stdout:8 2024-05-08T16:31:20.141 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:31:20.141 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-08T16:31:20.600 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:31:21.602 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:31:21.602 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:31:21.602 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-05-08T16:31:21.602 INFO:tasks.workunit.client.0.smithi172.stdout:9 2024-05-08T16:31:21.603 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:31:21.603 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-08T16:31:22.149 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:31:23.151 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:31:23.151 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:31:23.151 INFO:tasks.workunit.client.0.smithi172.stdout:10 2024-05-08T16:31:23.152 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-05-08T16:31:23.152 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:31:23.152 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-08T16:31:23.718 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:31:24.720 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:31:24.720 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:31:24.721 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-05-08T16:31:24.721 INFO:tasks.workunit.client.0.smithi172.stdout:11 2024-05-08T16:31:24.721 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:31:24.721 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-08T16:31:25.507 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:31:26.509 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:31:26.509 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:31:26.509 INFO:tasks.workunit.client.0.smithi172.stdout:12 2024-05-08T16:31:26.510 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-05-08T16:31:26.510 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:31:26.510 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-08T16:31:27.116 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:31:28.117 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:31:28.118 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:31:28.118 INFO:tasks.workunit.client.0.smithi172.stdout:13 2024-05-08T16:31:28.118 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-05-08T16:31:28.118 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:31:28.118 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-08T16:31:28.791 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:31:29.793 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:31:29.793 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:31:29.793 INFO:tasks.workunit.client.0.smithi172.stdout:14 2024-05-08T16:31:29.794 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-05-08T16:31:29.794 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:31:29.794 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-08T16:31:30.577 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:31:31.579 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:31:31.579 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:31:31.579 INFO:tasks.workunit.client.0.smithi172.stdout:15 2024-05-08T16:31:31.580 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-05-08T16:31:31.580 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:31:31.580 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-08T16:31:32.888 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:31:33.890 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:31:33.890 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:31:33.891 INFO:tasks.workunit.client.0.smithi172.stdout:16 2024-05-08T16:31:33.891 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-05-08T16:31:33.891 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:31:33.891 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-08T16:31:34.869 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:31:35.871 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:31:35.871 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:31:35.871 INFO:tasks.workunit.client.0.smithi172.stdout:17 2024-05-08T16:31:35.872 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-05-08T16:31:35.872 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:31:35.872 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-08T16:31:36.716 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:31:37.717 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:31:37.717 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:31:37.718 INFO:tasks.workunit.client.0.smithi172.stdout:18 2024-05-08T16:31:37.718 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-05-08T16:31:37.718 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:31:37.718 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-08T16:31:38.579 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:31:39.581 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:31:39.581 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:31:39.581 INFO:tasks.workunit.client.0.smithi172.stdout:19 2024-05-08T16:31:39.582 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-05-08T16:31:39.582 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:31:39.582 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-08T16:31:40.206 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:31:41.210 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:31:41.210 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:31:41.210 INFO:tasks.workunit.client.0.smithi172.stdout:20 2024-05-08T16:31:41.210 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-05-08T16:31:41.211 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:31:41.211 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-08T16:31:41.787 INFO:tasks.workunit.client.0.smithi172.stdout:osd.4 up in weight 1 up_from 38 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/1587661565,v1:127.0.0.1:6835/1587661565] [v2:127.0.0.1:6836/1587661565,v1:127.0.0.1:6837/1587661565] exists,up 21f94280-e9a5-4de7-88e6-9d4371bc9808 2024-05-08T16:31:41.789 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-08T16:31:41.789 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-08T16:31:41.789 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-08T16:31:41.789 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-05-08T16:31:41.789 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 5 2024-05-08T16:31:41.789 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-05-08T16:31:41.790 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-05-08T16:31:41.790 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=5 2024-05-08T16:31:41.790 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-05-08T16:31:41.790 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/5 2024-05-08T16:31:41.790 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=4457b0f1-028b-482b-8ee3-7ee05861e6cc --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-05-08T16:31:41.790 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-08T16:31:41.790 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-05-08T16:31:41.790 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-08T16:31:41.790 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/5' 2024-05-08T16:31:41.790 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/5/journal' 2024-05-08T16:31:41.790 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-05-08T16:31:41.790 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-05-08T16:31:41.790 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-05-08T16:31:41.791 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-05-08T16:31:41.791 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-08T16:31:41.791 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-08T16:31:41.791 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-08T16:31:41.791 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T16:31:41.791 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81110 2024-05-08T16:31:41.792 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.81110/$cluster-$name.asok' 2024-05-08T16:31:41.792 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.81110/$cluster-$name.asok' 2024-05-08T16:31:41.792 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-05-08T16:31:41.792 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-05-08T16:31:41.792 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-05-08T16:31:41.793 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-05-08T16:31:41.793 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-05-08T16:31:41.793 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-08T16:31:41.793 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-08T16:31:41.793 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-08T16:31:41.793 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-08T16:31:41.793 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-05-08T16:31:41.793 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-05-08T16:31:41.793 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/5 2024-05-08T16:31:41.793 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-05-08T16:31:41.793 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=4df262cc-b464-44f3-a189-ac75e00f79b2 2024-05-08T16:31:41.794 INFO:tasks.workunit.client.0.smithi172.stdout:add osd5 4df262cc-b464-44f3-a189-ac75e00f79b2 2024-05-08T16:31:41.794 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd5 4df262cc-b464-44f3-a189-ac75e00f79b2' 2024-05-08T16:31:41.794 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-05-08T16:31:41.816 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDtqDtmLij/MBAA4OEz42n1lS62Fy0z1ZJVsw== 2024-05-08T16:31:41.816 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDtqDtmLij/MBAA4OEz42n1lS62Fy0z1ZJVsw=="}' 2024-05-08T16:31:41.816 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 4df262cc-b464-44f3-a189-ac75e00f79b2 -i td/test-erasure-eio/5/new.json 2024-05-08T16:31:42.790 INFO:tasks.workunit.client.0.smithi172.stdout:5 2024-05-08T16:31:42.810 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/5/new.json 2024-05-08T16:31:42.811 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 5 --fsid=4457b0f1-028b-482b-8ee3-7ee05861e6cc --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/5 --osd-journal=td/test-erasure-eio/5/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81110/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDtqDtmLij/MBAA4OEz42n1lS62Fy0z1ZJVsw== --osd-uuid 4df262cc-b464-44f3-a189-ac75e00f79b2 2024-05-08T16:31:42.846 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:31:42.849+0000 7feca92b37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:31:43.019 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:31:43.021+0000 7feca92b37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:31:43.021 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:31:43.025+0000 7feca92b37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:31:43.022 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:31:43.025+0000 7feca92b37c0 -1 bluestore(td/test-erasure-eio/5/block) _read_bdev_label failed to open td/test-erasure-eio/5/block: (2) No such file or directory 2024-05-08T16:31:43.022 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:31:43.025+0000 7feca92b37c0 -1 bluestore(td/test-erasure-eio/5/block) _read_bdev_label failed to open td/test-erasure-eio/5/block: (2) No such file or directory 2024-05-08T16:31:43.023 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:31:43.025+0000 7feca92b37c0 -1 bluestore(td/test-erasure-eio/5/block) _read_bdev_label failed to open td/test-erasure-eio/5/block: (2) No such file or directory 2024-05-08T16:31:43.211 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:31:43.213+0000 7feca92b37c0 -1 bluestore(td/test-erasure-eio/5) _read_fsid unparsable uuid 2024-05-08T16:31:50.309 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/5/keyring 2024-05-08T16:31:50.309 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-05-08T16:31:50.310 INFO:tasks.workunit.client.0.smithi172.stdout:adding osd5 key to auth repository 2024-05-08T16:31:50.311 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd5 key to auth repository 2024-05-08T16:31:50.311 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/5/keyring auth add osd.5 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-05-08T16:31:50.882 INFO:tasks.workunit.client.0.smithi172.stdout:start osd.5 2024-05-08T16:31:50.883 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.5 2024-05-08T16:31:50.883 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 5 --fsid=4457b0f1-028b-482b-8ee3-7ee05861e6cc --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/5 --osd-journal=td/test-erasure-eio/5/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81110/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-05-08T16:31:50.883 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-05-08T16:31:50.883 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-05-08T16:31:50.883 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-05-08T16:31:50.913 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:31:50.917+0000 7fb29bead7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:31:50.916 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:31:50.917+0000 7fb29bead7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:31:50.917 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:31:50.921+0000 7fb29bead7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:31:51.327 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 5 2024-05-08T16:31:51.327 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-08T16:31:51.327 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=5 2024-05-08T16:31:51.327 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-08T16:31:51.327 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-08T16:31:51.327 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:31:51.327 INFO:tasks.workunit.client.0.smithi172.stdout:0 2024-05-08T16:31:51.328 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-08T16:31:51.328 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:31:51.328 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-05-08T16:31:51.490 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:31:51.493+0000 7fb29bead7c0 -1 Falling back to public interface 2024-05-08T16:31:51.759 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:31:52.761 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:31:52.761 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:31:52.761 INFO:tasks.workunit.client.0.smithi172.stdout:1 2024-05-08T16:31:52.761 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-08T16:31:52.762 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:31:52.762 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-05-08T16:31:53.180 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:31:54.109 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:31:54.113+0000 7fb29bead7c0 -1 osd.5 0 log_to_monitors true 2024-05-08T16:31:54.181 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:31:54.181 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:31:54.182 INFO:tasks.workunit.client.0.smithi172.stdout:2 2024-05-08T16:31:54.182 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-08T16:31:54.182 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:31:54.182 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-05-08T16:31:55.128 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:31:56.130 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:31:56.130 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:31:56.130 INFO:tasks.workunit.client.0.smithi172.stdout:3 2024-05-08T16:31:56.131 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-05-08T16:31:56.131 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:31:56.131 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-05-08T16:31:56.865 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:31:57.867 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:31:57.867 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:31:57.868 INFO:tasks.workunit.client.0.smithi172.stdout:4 2024-05-08T16:31:57.868 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-05-08T16:31:57.868 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:31:57.868 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-05-08T16:31:58.481 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:31:59.482 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:31:59.482 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:31:59.483 INFO:tasks.workunit.client.0.smithi172.stdout:5 2024-05-08T16:31:59.483 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-05-08T16:31:59.483 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:31:59.483 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-05-08T16:31:59.905 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:32:00.907 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:32:00.907 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:32:00.907 INFO:tasks.workunit.client.0.smithi172.stdout:6 2024-05-08T16:32:00.908 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-05-08T16:32:00.908 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:32:00.908 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-05-08T16:32:01.685 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:32:02.687 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:32:02.687 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:32:02.687 INFO:tasks.workunit.client.0.smithi172.stdout:7 2024-05-08T16:32:02.688 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-05-08T16:32:02.688 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:32:02.688 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-05-08T16:32:03.402 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:32:04.404 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:32:04.404 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:32:04.405 INFO:tasks.workunit.client.0.smithi172.stdout:8 2024-05-08T16:32:04.405 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-05-08T16:32:04.405 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:32:04.405 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-05-08T16:32:04.854 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:32:05.856 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:32:05.856 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:32:05.856 INFO:tasks.workunit.client.0.smithi172.stdout:9 2024-05-08T16:32:05.857 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-05-08T16:32:05.857 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:32:05.857 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-05-08T16:32:06.437 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:32:07.439 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:32:07.439 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:32:07.439 INFO:tasks.workunit.client.0.smithi172.stdout:10 2024-05-08T16:32:07.440 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-05-08T16:32:07.440 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:32:07.440 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-05-08T16:32:07.880 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:32:08.882 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:32:08.882 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:32:08.882 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-05-08T16:32:08.883 INFO:tasks.workunit.client.0.smithi172.stdout:11 2024-05-08T16:32:08.883 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:32:08.883 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-05-08T16:32:09.448 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:32:10.449 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:32:10.450 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:32:10.450 INFO:tasks.workunit.client.0.smithi172.stdout:12 2024-05-08T16:32:10.451 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-05-08T16:32:10.451 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:32:10.451 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-05-08T16:32:11.134 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:32:12.136 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:32:12.136 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:32:12.137 INFO:tasks.workunit.client.0.smithi172.stdout:13 2024-05-08T16:32:12.137 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-05-08T16:32:12.137 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:32:12.137 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-05-08T16:32:13.026 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:32:14.028 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:32:14.028 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:32:14.028 INFO:tasks.workunit.client.0.smithi172.stdout:14 2024-05-08T16:32:14.029 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-05-08T16:32:14.029 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:32:14.029 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-05-08T16:32:15.018 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:32:16.020 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:32:16.020 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:32:16.020 INFO:tasks.workunit.client.0.smithi172.stdout:15 2024-05-08T16:32:16.021 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-05-08T16:32:16.021 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:32:16.021 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-05-08T16:32:17.091 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:32:18.093 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:32:18.093 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:32:18.093 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-05-08T16:32:18.094 INFO:tasks.workunit.client.0.smithi172.stdout:16 2024-05-08T16:32:18.094 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:32:18.094 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-05-08T16:32:19.002 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:32:20.004 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:32:20.004 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:32:20.004 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-05-08T16:32:20.005 INFO:tasks.workunit.client.0.smithi172.stdout:17 2024-05-08T16:32:20.005 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:32:20.005 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-05-08T16:32:20.768 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:32:21.769 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:32:21.769 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:32:21.770 INFO:tasks.workunit.client.0.smithi172.stdout:18 2024-05-08T16:32:21.770 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-05-08T16:32:21.770 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:32:21.770 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-05-08T16:32:22.620 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:32:23.621 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:32:23.622 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:32:23.622 INFO:tasks.workunit.client.0.smithi172.stdout:19 2024-05-08T16:32:23.622 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-05-08T16:32:23.623 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:32:23.623 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-05-08T16:32:24.225 INFO:tasks.workunit.client.0.smithi172.stdout:osd.5 up in weight 1 up_from 48 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6842/3426266244,v1:127.0.0.1:6843/3426266244] [v2:127.0.0.1:6844/3426266244,v1:127.0.0.1:6845/3426266244] exists,up 4df262cc-b464-44f3-a189-ac75e00f79b2 2024-05-08T16:32:24.231 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-08T16:32:24.232 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-08T16:32:24.232 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-08T16:32:24.232 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-05-08T16:32:24.232 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 6 2024-05-08T16:32:24.232 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-05-08T16:32:24.232 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-05-08T16:32:24.232 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=6 2024-05-08T16:32:24.232 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-05-08T16:32:24.233 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/6 2024-05-08T16:32:24.233 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=4457b0f1-028b-482b-8ee3-7ee05861e6cc --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-05-08T16:32:24.233 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-08T16:32:24.233 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-05-08T16:32:24.233 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-08T16:32:24.233 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/6' 2024-05-08T16:32:24.233 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/6/journal' 2024-05-08T16:32:24.233 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-05-08T16:32:24.234 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-05-08T16:32:24.234 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-05-08T16:32:24.234 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-05-08T16:32:24.234 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-08T16:32:24.234 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-08T16:32:24.235 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-08T16:32:24.235 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T16:32:24.235 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81110 2024-05-08T16:32:24.235 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.81110/$cluster-$name.asok' 2024-05-08T16:32:24.235 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.81110/$cluster-$name.asok' 2024-05-08T16:32:24.235 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-05-08T16:32:24.236 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-05-08T16:32:24.236 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-05-08T16:32:24.236 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-05-08T16:32:24.236 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-05-08T16:32:24.236 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-08T16:32:24.236 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-08T16:32:24.236 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-08T16:32:24.237 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-08T16:32:24.237 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-05-08T16:32:24.237 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-05-08T16:32:24.237 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/6 2024-05-08T16:32:24.237 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-05-08T16:32:24.237 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=ba564330-c793-400a-8f0f-6748ad6d8265 2024-05-08T16:32:24.237 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd6 ba564330-c793-400a-8f0f-6748ad6d8265' 2024-05-08T16:32:24.238 INFO:tasks.workunit.client.0.smithi172.stdout:add osd6 ba564330-c793-400a-8f0f-6748ad6d8265 2024-05-08T16:32:24.238 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-05-08T16:32:24.252 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAYqTtm4w9nDxAAhnJ5RUzU0WEJ4uR32N4HkA== 2024-05-08T16:32:24.252 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAYqTtm4w9nDxAAhnJ5RUzU0WEJ4uR32N4HkA=="}' 2024-05-08T16:32:24.252 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new ba564330-c793-400a-8f0f-6748ad6d8265 -i td/test-erasure-eio/6/new.json 2024-05-08T16:32:25.193 INFO:tasks.workunit.client.0.smithi172.stdout:6 2024-05-08T16:32:25.217 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/6/new.json 2024-05-08T16:32:25.219 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 6 --fsid=4457b0f1-028b-482b-8ee3-7ee05861e6cc --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/6 --osd-journal=td/test-erasure-eio/6/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81110/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAYqTtm4w9nDxAAhnJ5RUzU0WEJ4uR32N4HkA== --osd-uuid ba564330-c793-400a-8f0f-6748ad6d8265 2024-05-08T16:32:25.254 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:32:25.256+0000 7f4a624407c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:32:25.256 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:32:25.256+0000 7f4a624407c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:32:25.257 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:32:25.260+0000 7f4a624407c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:32:25.257 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:32:25.260+0000 7f4a624407c0 -1 bluestore(td/test-erasure-eio/6/block) _read_bdev_label failed to open td/test-erasure-eio/6/block: (2) No such file or directory 2024-05-08T16:32:25.258 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:32:25.260+0000 7f4a624407c0 -1 bluestore(td/test-erasure-eio/6/block) _read_bdev_label failed to open td/test-erasure-eio/6/block: (2) No such file or directory 2024-05-08T16:32:25.258 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:32:25.260+0000 7f4a624407c0 -1 bluestore(td/test-erasure-eio/6/block) _read_bdev_label failed to open td/test-erasure-eio/6/block: (2) No such file or directory 2024-05-08T16:32:25.385 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:32:25.384+0000 7f4a624407c0 -1 bluestore(td/test-erasure-eio/6) _read_fsid unparsable uuid 2024-05-08T16:32:32.665 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/6/keyring 2024-05-08T16:32:32.665 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-05-08T16:32:32.666 INFO:tasks.workunit.client.0.smithi172.stdout:adding osd6 key to auth repository 2024-05-08T16:32:32.667 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd6 key to auth repository 2024-05-08T16:32:32.667 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/6/keyring auth add osd.6 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-05-08T16:32:33.227 INFO:tasks.workunit.client.0.smithi172.stdout:start osd.6 2024-05-08T16:32:33.228 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.6 2024-05-08T16:32:33.228 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 6 --fsid=4457b0f1-028b-482b-8ee3-7ee05861e6cc --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/6 --osd-journal=td/test-erasure-eio/6/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81110/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-05-08T16:32:33.228 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-05-08T16:32:33.228 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-05-08T16:32:33.228 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-05-08T16:32:33.258 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:32:33.260+0000 7ff2b36df7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:32:33.260 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:32:33.264+0000 7ff2b36df7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:32:33.262 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:32:33.264+0000 7ff2b36df7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:32:33.671 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 6 2024-05-08T16:32:33.671 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-08T16:32:33.671 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=6 2024-05-08T16:32:33.671 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-08T16:32:33.671 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-08T16:32:33.671 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:32:33.671 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-08T16:32:33.672 INFO:tasks.workunit.client.0.smithi172.stdout:0 2024-05-08T16:32:33.672 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:32:33.672 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-05-08T16:32:33.833 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:32:33.836+0000 7ff2b36df7c0 -1 Falling back to public interface 2024-05-08T16:32:34.099 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:32:35.101 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:32:35.101 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:32:35.101 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-08T16:32:35.101 INFO:tasks.workunit.client.0.smithi172.stdout:1 2024-05-08T16:32:35.102 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:32:35.102 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-05-08T16:32:35.511 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:32:36.506 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:32:36.508+0000 7ff2b36df7c0 -1 osd.6 0 log_to_monitors true 2024-05-08T16:32:36.513 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:32:36.513 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:32:36.513 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-08T16:32:36.513 INFO:tasks.workunit.client.0.smithi172.stdout:2 2024-05-08T16:32:36.513 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-05-08T16:32:36.514 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:32:37.095 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:32:38.097 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:32:38.097 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:32:38.098 INFO:tasks.workunit.client.0.smithi172.stdout:3 2024-05-08T16:32:38.098 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-05-08T16:32:38.098 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:32:38.098 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-05-08T16:32:38.614 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:32:39.616 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:32:39.616 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:32:39.616 INFO:tasks.workunit.client.0.smithi172.stdout:4 2024-05-08T16:32:39.617 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-05-08T16:32:39.617 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:32:39.617 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-05-08T16:32:40.323 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:32:41.325 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:32:41.325 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:32:41.326 INFO:tasks.workunit.client.0.smithi172.stdout:5 2024-05-08T16:32:41.326 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-05-08T16:32:41.326 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:32:41.326 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-05-08T16:32:42.005 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:32:43.007 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:32:43.007 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:32:43.007 INFO:tasks.workunit.client.0.smithi172.stdout:6 2024-05-08T16:32:43.008 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-05-08T16:32:43.008 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:32:43.008 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-05-08T16:32:43.615 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:32:44.617 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:32:44.617 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:32:44.617 INFO:tasks.workunit.client.0.smithi172.stdout:7 2024-05-08T16:32:44.618 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-05-08T16:32:44.618 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:32:44.618 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-05-08T16:32:45.062 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:32:46.064 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:32:46.064 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:32:46.065 INFO:tasks.workunit.client.0.smithi172.stdout:8 2024-05-08T16:32:46.065 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-05-08T16:32:46.065 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:32:46.065 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-05-08T16:32:46.601 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:32:47.603 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:32:47.603 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:32:47.603 INFO:tasks.workunit.client.0.smithi172.stdout:9 2024-05-08T16:32:47.604 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-05-08T16:32:47.604 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:32:47.604 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-05-08T16:32:48.025 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:32:49.027 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:32:49.027 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:32:49.027 INFO:tasks.workunit.client.0.smithi172.stdout:10 2024-05-08T16:32:49.028 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-05-08T16:32:49.028 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:32:49.028 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-05-08T16:32:49.695 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:32:50.697 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:32:50.697 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:32:50.697 INFO:tasks.workunit.client.0.smithi172.stdout:11 2024-05-08T16:32:50.698 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-05-08T16:32:50.698 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:32:50.698 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-05-08T16:32:51.242 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:32:52.244 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:32:52.244 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:32:52.244 INFO:tasks.workunit.client.0.smithi172.stdout:12 2024-05-08T16:32:52.245 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-05-08T16:32:52.245 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:32:52.245 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-05-08T16:32:53.133 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:32:54.134 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:32:54.134 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:32:54.135 INFO:tasks.workunit.client.0.smithi172.stdout:13 2024-05-08T16:32:54.135 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-05-08T16:32:54.135 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:32:54.135 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-05-08T16:32:54.968 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:32:55.970 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:32:55.970 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:32:55.971 INFO:tasks.workunit.client.0.smithi172.stdout:14 2024-05-08T16:32:55.971 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-05-08T16:32:55.971 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:32:55.971 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-05-08T16:32:56.774 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:32:57.776 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:32:57.776 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:32:57.776 INFO:tasks.workunit.client.0.smithi172.stdout:15 2024-05-08T16:32:57.777 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-05-08T16:32:57.777 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:32:57.777 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-05-08T16:32:58.497 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:32:59.499 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:32:59.499 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:32:59.500 INFO:tasks.workunit.client.0.smithi172.stdout:16 2024-05-08T16:32:59.501 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-05-08T16:32:59.501 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:32:59.501 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-05-08T16:33:00.788 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:33:01.789 INFO:tasks.workunit.client.0.smithi172.stdout:17 2024-05-08T16:33:01.790 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:33:01.791 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:33:01.791 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-05-08T16:33:01.791 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:33:01.791 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-05-08T16:33:02.542 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:33:03.544 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:33:03.544 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:33:03.545 INFO:tasks.workunit.client.0.smithi172.stdout:18 2024-05-08T16:33:03.545 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-05-08T16:33:03.545 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:33:03.545 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-05-08T16:33:04.017 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:33:05.018 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:33:05.019 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:33:05.019 INFO:tasks.workunit.client.0.smithi172.stdout:19 2024-05-08T16:33:05.019 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-05-08T16:33:05.020 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:33:05.020 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-05-08T16:33:05.695 INFO:tasks.workunit.client.0.smithi172.stdout:osd.6 up in weight 1 up_from 56 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6850/1529041613,v1:127.0.0.1:6851/1529041613] [v2:127.0.0.1:6852/1529041613,v1:127.0.0.1:6853/1529041613] exists,up ba564330-c793-400a-8f0f-6748ad6d8265 2024-05-08T16:33:05.699 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-08T16:33:05.699 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-08T16:33:05.699 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-08T16:33:05.699 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-05-08T16:33:05.699 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 7 2024-05-08T16:33:05.699 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-05-08T16:33:05.699 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-05-08T16:33:05.699 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=7 2024-05-08T16:33:05.699 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-05-08T16:33:05.699 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/7 2024-05-08T16:33:05.699 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=4457b0f1-028b-482b-8ee3-7ee05861e6cc --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-05-08T16:33:05.700 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-08T16:33:05.700 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-05-08T16:33:05.700 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-08T16:33:05.700 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/7' 2024-05-08T16:33:05.700 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/7/journal' 2024-05-08T16:33:05.700 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-05-08T16:33:05.700 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-05-08T16:33:05.700 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-05-08T16:33:05.700 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-05-08T16:33:05.700 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-08T16:33:05.700 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-08T16:33:05.700 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-08T16:33:05.700 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T16:33:05.701 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81110 2024-05-08T16:33:05.701 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.81110/$cluster-$name.asok' 2024-05-08T16:33:05.701 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.81110/$cluster-$name.asok' 2024-05-08T16:33:05.701 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-05-08T16:33:05.701 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-05-08T16:33:05.701 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-05-08T16:33:05.701 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-05-08T16:33:05.701 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-05-08T16:33:05.701 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-08T16:33:05.701 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-08T16:33:05.701 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-08T16:33:05.701 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-08T16:33:05.701 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-05-08T16:33:05.702 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-05-08T16:33:05.702 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/7 2024-05-08T16:33:05.702 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-05-08T16:33:05.702 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=99bf73e6-8c93-4d3b-adb0-d28607dec22a 2024-05-08T16:33:05.703 INFO:tasks.workunit.client.0.smithi172.stdout:add osd7 99bf73e6-8c93-4d3b-adb0-d28607dec22a 2024-05-08T16:33:05.703 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd7 99bf73e6-8c93-4d3b-adb0-d28607dec22a' 2024-05-08T16:33:05.703 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-05-08T16:33:05.721 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBBqTtmbz1aKxAAiiNO0MFPuyDGoPtU+pFdug== 2024-05-08T16:33:05.721 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBBqTtmbz1aKxAAiiNO0MFPuyDGoPtU+pFdug=="}' 2024-05-08T16:33:05.721 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 99bf73e6-8c93-4d3b-adb0-d28607dec22a -i td/test-erasure-eio/7/new.json 2024-05-08T16:33:07.062 INFO:tasks.workunit.client.0.smithi172.stdout:7 2024-05-08T16:33:07.082 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/7/new.json 2024-05-08T16:33:07.083 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 7 --fsid=4457b0f1-028b-482b-8ee3-7ee05861e6cc --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/7 --osd-journal=td/test-erasure-eio/7/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81110/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBBqTtmbz1aKxAAiiNO0MFPuyDGoPtU+pFdug== --osd-uuid 99bf73e6-8c93-4d3b-adb0-d28607dec22a 2024-05-08T16:33:07.118 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:33:07.119+0000 7fa8d669c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:33:07.152 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:33:07.151+0000 7fa8d669c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:33:07.154 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:33:07.155+0000 7fa8d669c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:33:07.154 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:33:07.155+0000 7fa8d669c7c0 -1 bluestore(td/test-erasure-eio/7/block) _read_bdev_label failed to open td/test-erasure-eio/7/block: (2) No such file or directory 2024-05-08T16:33:07.154 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:33:07.155+0000 7fa8d669c7c0 -1 bluestore(td/test-erasure-eio/7/block) _read_bdev_label failed to open td/test-erasure-eio/7/block: (2) No such file or directory 2024-05-08T16:33:07.154 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:33:07.155+0000 7fa8d669c7c0 -1 bluestore(td/test-erasure-eio/7/block) _read_bdev_label failed to open td/test-erasure-eio/7/block: (2) No such file or directory 2024-05-08T16:33:07.274 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:33:07.275+0000 7fa8d669c7c0 -1 bluestore(td/test-erasure-eio/7) _read_fsid unparsable uuid 2024-05-08T16:33:14.536 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/7/keyring 2024-05-08T16:33:14.537 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-05-08T16:33:14.538 INFO:tasks.workunit.client.0.smithi172.stdout:adding osd7 key to auth repository 2024-05-08T16:33:14.538 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd7 key to auth repository 2024-05-08T16:33:14.538 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/7/keyring auth add osd.7 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-05-08T16:33:15.126 INFO:tasks.workunit.client.0.smithi172.stdout:start osd.7 2024-05-08T16:33:15.127 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.7 2024-05-08T16:33:15.127 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-05-08T16:33:15.128 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-05-08T16:33:15.128 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 7 --fsid=4457b0f1-028b-482b-8ee3-7ee05861e6cc --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/7 --osd-journal=td/test-erasure-eio/7/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81110/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-05-08T16:33:15.128 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-05-08T16:33:15.158 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:33:15.163+0000 7f518d3be7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:33:15.160 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:33:15.167+0000 7f518d3be7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:33:15.162 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:33:15.167+0000 7f518d3be7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:33:15.558 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 7 2024-05-08T16:33:15.558 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-08T16:33:15.558 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=7 2024-05-08T16:33:15.558 INFO:tasks.workunit.client.0.smithi172.stdout:0 2024-05-08T16:33:15.559 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-08T16:33:15.559 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-08T16:33:15.559 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:33:15.559 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-08T16:33:15.559 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-05-08T16:33:15.559 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:33:15.705 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:33:15.711+0000 7f518d3be7c0 -1 Falling back to public interface 2024-05-08T16:33:15.987 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:33:16.989 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:33:16.989 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:33:16.989 INFO:tasks.workunit.client.0.smithi172.stdout:1 2024-05-08T16:33:16.989 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-08T16:33:16.990 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:33:16.990 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-05-08T16:33:17.420 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:33:18.360 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:33:18.367+0000 7f518d3be7c0 -1 osd.7 0 log_to_monitors true 2024-05-08T16:33:18.422 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:33:18.422 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:33:18.422 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-08T16:33:18.422 INFO:tasks.workunit.client.0.smithi172.stdout:2 2024-05-08T16:33:18.423 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:33:18.423 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-05-08T16:33:18.998 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:33:19.999 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:33:20.000 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:33:20.000 INFO:tasks.workunit.client.0.smithi172.stdout:3 2024-05-08T16:33:20.000 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-05-08T16:33:20.000 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:33:20.001 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-05-08T16:33:20.781 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:33:21.782 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:33:21.782 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:33:21.783 INFO:tasks.workunit.client.0.smithi172.stdout:4 2024-05-08T16:33:21.783 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-05-08T16:33:21.783 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:33:21.783 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-05-08T16:33:22.468 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:33:23.469 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:33:23.469 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:33:23.470 INFO:tasks.workunit.client.0.smithi172.stdout:5 2024-05-08T16:33:23.470 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-05-08T16:33:23.470 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:33:23.470 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-05-08T16:33:24.097 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:33:25.099 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:33:25.099 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:33:25.100 INFO:tasks.workunit.client.0.smithi172.stdout:6 2024-05-08T16:33:25.100 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-05-08T16:33:25.100 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:33:25.100 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-05-08T16:33:25.750 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:33:26.752 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:33:26.752 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:33:26.752 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-05-08T16:33:26.753 INFO:tasks.workunit.client.0.smithi172.stdout:7 2024-05-08T16:33:26.753 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:33:26.753 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-05-08T16:33:27.345 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:33:28.347 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:33:28.347 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:33:28.348 INFO:tasks.workunit.client.0.smithi172.stdout:8 2024-05-08T16:33:28.348 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-05-08T16:33:28.348 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:33:28.348 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-05-08T16:33:28.767 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:33:29.769 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:33:29.769 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:33:29.769 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-05-08T16:33:29.770 INFO:tasks.workunit.client.0.smithi172.stdout:9 2024-05-08T16:33:29.770 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:33:29.770 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-05-08T16:33:30.382 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:33:31.384 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:33:31.384 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:33:31.384 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-05-08T16:33:31.384 INFO:tasks.workunit.client.0.smithi172.stdout:10 2024-05-08T16:33:31.385 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:33:31.385 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-05-08T16:33:31.832 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:33:32.834 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:33:32.835 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:33:32.835 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-05-08T16:33:32.835 INFO:tasks.workunit.client.0.smithi172.stdout:11 2024-05-08T16:33:32.835 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:33:32.835 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-05-08T16:33:33.400 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:33:34.402 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:33:34.402 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:33:34.402 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-05-08T16:33:34.402 INFO:tasks.workunit.client.0.smithi172.stdout:12 2024-05-08T16:33:34.402 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:33:34.403 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-05-08T16:33:34.993 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:33:35.995 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:33:35.995 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:33:35.995 INFO:tasks.workunit.client.0.smithi172.stdout:13 2024-05-08T16:33:35.996 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-05-08T16:33:35.996 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:33:35.996 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-05-08T16:33:37.216 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:33:38.217 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:33:38.218 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:33:38.218 INFO:tasks.workunit.client.0.smithi172.stdout:14 2024-05-08T16:33:38.218 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-05-08T16:33:38.218 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:33:38.219 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-05-08T16:33:38.964 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:33:39.966 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:33:39.966 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:33:39.967 INFO:tasks.workunit.client.0.smithi172.stdout:15 2024-05-08T16:33:39.967 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-05-08T16:33:39.967 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:33:39.967 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-05-08T16:33:40.631 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:33:41.632 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:33:41.633 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:33:41.633 INFO:tasks.workunit.client.0.smithi172.stdout:16 2024-05-08T16:33:41.633 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-05-08T16:33:41.633 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:33:41.634 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-05-08T16:33:42.733 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:33:43.735 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:33:43.735 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:33:43.736 INFO:tasks.workunit.client.0.smithi172.stdout:17 2024-05-08T16:33:43.736 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-05-08T16:33:43.736 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:33:43.736 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-05-08T16:33:44.620 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:33:45.622 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:33:45.623 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:33:45.623 INFO:tasks.workunit.client.0.smithi172.stdout:18 2024-05-08T16:33:45.623 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-05-08T16:33:45.623 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:33:45.624 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-05-08T16:33:47.130 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:33:48.132 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:33:48.132 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:33:48.133 INFO:tasks.workunit.client.0.smithi172.stdout:19 2024-05-08T16:33:48.133 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-05-08T16:33:48.133 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:33:48.133 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.7 up' 2024-05-08T16:33:48.608 INFO:tasks.workunit.client.0.smithi172.stdout:osd.7 up in weight 1 up_from 66 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6858/3714178474,v1:127.0.0.1:6859/3714178474] [v2:127.0.0.1:6860/3714178474,v1:127.0.0.1:6861/3714178474] exists,up 99bf73e6-8c93-4d3b-adb0-d28607dec22a 2024-05-08T16:33:48.612 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-08T16:33:48.612 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-08T16:33:48.612 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-08T16:33:48.612 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-05-08T16:33:48.612 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 8 2024-05-08T16:33:48.612 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-05-08T16:33:48.612 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-05-08T16:33:48.612 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=8 2024-05-08T16:33:48.612 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-05-08T16:33:48.612 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/8 2024-05-08T16:33:48.613 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=4457b0f1-028b-482b-8ee3-7ee05861e6cc --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-05-08T16:33:48.613 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-08T16:33:48.613 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-05-08T16:33:48.613 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-08T16:33:48.613 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/8' 2024-05-08T16:33:48.613 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/8/journal' 2024-05-08T16:33:48.613 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-05-08T16:33:48.613 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-05-08T16:33:48.613 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-05-08T16:33:48.613 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-05-08T16:33:48.613 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-08T16:33:48.613 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-08T16:33:48.613 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-08T16:33:48.613 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T16:33:48.613 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81110 2024-05-08T16:33:48.614 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.81110/$cluster-$name.asok' 2024-05-08T16:33:48.614 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.81110/$cluster-$name.asok' 2024-05-08T16:33:48.614 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-05-08T16:33:48.614 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-05-08T16:33:48.614 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-05-08T16:33:48.614 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-05-08T16:33:48.614 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-05-08T16:33:48.614 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-08T16:33:48.614 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-08T16:33:48.614 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-08T16:33:48.614 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-08T16:33:48.614 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-05-08T16:33:48.614 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-05-08T16:33:48.614 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/8 2024-05-08T16:33:48.614 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-05-08T16:33:48.615 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=e6c35a5a-8838-46eb-8a61-62016f175053 2024-05-08T16:33:48.615 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd8 e6c35a5a-8838-46eb-8a61-62016f175053' 2024-05-08T16:33:48.615 INFO:tasks.workunit.client.0.smithi172.stdout:add osd8 e6c35a5a-8838-46eb-8a61-62016f175053 2024-05-08T16:33:48.616 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-05-08T16:33:48.632 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBsqTtmdlsEJhAAogVRnxrktxHvkCBXSOs7kg== 2024-05-08T16:33:48.632 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBsqTtmdlsEJhAAogVRnxrktxHvkCBXSOs7kg=="}' 2024-05-08T16:33:48.632 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new e6c35a5a-8838-46eb-8a61-62016f175053 -i td/test-erasure-eio/8/new.json 2024-05-08T16:33:49.370 INFO:tasks.workunit.client.0.smithi172.stdout:8 2024-05-08T16:33:49.388 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/8/new.json 2024-05-08T16:33:49.389 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 8 --fsid=4457b0f1-028b-482b-8ee3-7ee05861e6cc --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/8 --osd-journal=td/test-erasure-eio/8/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81110/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBsqTtmdlsEJhAAogVRnxrktxHvkCBXSOs7kg== --osd-uuid e6c35a5a-8838-46eb-8a61-62016f175053 2024-05-08T16:33:49.429 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:33:49.434+0000 7fbecfd227c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:33:49.519 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:33:49.522+0000 7fbecfd227c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:33:49.521 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:33:49.526+0000 7fbecfd227c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:33:49.521 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:33:49.526+0000 7fbecfd227c0 -1 bluestore(td/test-erasure-eio/8/block) _read_bdev_label failed to open td/test-erasure-eio/8/block: (2) No such file or directory 2024-05-08T16:33:49.521 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:33:49.526+0000 7fbecfd227c0 -1 bluestore(td/test-erasure-eio/8/block) _read_bdev_label failed to open td/test-erasure-eio/8/block: (2) No such file or directory 2024-05-08T16:33:49.522 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:33:49.526+0000 7fbecfd227c0 -1 bluestore(td/test-erasure-eio/8/block) _read_bdev_label failed to open td/test-erasure-eio/8/block: (2) No such file or directory 2024-05-08T16:33:49.676 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:33:49.682+0000 7fbecfd227c0 -1 bluestore(td/test-erasure-eio/8) _read_fsid unparsable uuid 2024-05-08T16:33:58.550 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/8/keyring 2024-05-08T16:33:58.550 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-05-08T16:33:58.551 INFO:tasks.workunit.client.0.smithi172.stdout:adding osd8 key to auth repository 2024-05-08T16:33:58.551 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd8 key to auth repository 2024-05-08T16:33:58.551 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/8/keyring auth add osd.8 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-05-08T16:33:59.110 INFO:tasks.workunit.client.0.smithi172.stdout:start osd.8 2024-05-08T16:33:59.111 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.8 2024-05-08T16:33:59.111 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-05-08T16:33:59.111 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 8 --fsid=4457b0f1-028b-482b-8ee3-7ee05861e6cc --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/8 --osd-journal=td/test-erasure-eio/8/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81110/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-05-08T16:33:59.111 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-05-08T16:33:59.111 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-05-08T16:33:59.143 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:33:59.146+0000 7fb343fdd7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:33:59.145 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:33:59.150+0000 7fb343fdd7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:33:59.146 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:33:59.150+0000 7fb343fdd7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:33:59.496 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 8 2024-05-08T16:33:59.496 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-08T16:33:59.496 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=8 2024-05-08T16:33:59.496 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-08T16:33:59.496 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-08T16:33:59.496 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:33:59.497 INFO:tasks.workunit.client.0.smithi172.stdout:0 2024-05-08T16:33:59.497 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-08T16:33:59.497 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:33:59.497 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-05-08T16:33:59.706 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:33:59.710+0000 7fb343fdd7c0 -1 Falling back to public interface 2024-05-08T16:33:59.925 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:34:00.927 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:34:00.927 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:34:00.927 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-08T16:34:00.927 INFO:tasks.workunit.client.0.smithi172.stdout:1 2024-05-08T16:34:00.928 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:34:00.928 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-05-08T16:34:01.354 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:34:02.355 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:34:02.355 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:34:02.355 INFO:tasks.workunit.client.0.smithi172.stdout:2 2024-05-08T16:34:02.356 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-08T16:34:02.356 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:34:02.356 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-05-08T16:34:02.364 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:34:02.366+0000 7fb343fdd7c0 -1 osd.8 0 log_to_monitors true 2024-05-08T16:34:02.785 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:34:03.786 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:34:03.787 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:34:03.787 INFO:tasks.workunit.client.0.smithi172.stdout:3 2024-05-08T16:34:03.787 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-05-08T16:34:03.787 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:34:03.788 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-05-08T16:34:04.795 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:34:05.797 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:34:05.797 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:34:05.797 INFO:tasks.workunit.client.0.smithi172.stdout:4 2024-05-08T16:34:05.798 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-05-08T16:34:05.798 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:34:05.798 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-05-08T16:34:06.391 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:34:07.393 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:34:07.393 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:34:07.393 INFO:tasks.workunit.client.0.smithi172.stdout:5 2024-05-08T16:34:07.394 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-05-08T16:34:07.394 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:34:07.394 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-05-08T16:34:08.013 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:34:09.015 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:34:09.015 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:34:09.015 INFO:tasks.workunit.client.0.smithi172.stdout:6 2024-05-08T16:34:09.016 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-05-08T16:34:09.029 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:34:09.029 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-05-08T16:34:09.505 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:34:10.507 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:34:10.507 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:34:10.507 INFO:tasks.workunit.client.0.smithi172.stdout:7 2024-05-08T16:34:10.507 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-05-08T16:34:10.508 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:34:10.508 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-05-08T16:34:10.955 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:34:11.957 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:34:11.957 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:34:11.958 INFO:tasks.workunit.client.0.smithi172.stdout:8 2024-05-08T16:34:11.958 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-05-08T16:34:11.959 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:34:11.959 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-05-08T16:34:12.449 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:34:13.450 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:34:13.451 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:34:13.451 INFO:tasks.workunit.client.0.smithi172.stdout:9 2024-05-08T16:34:13.451 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-05-08T16:34:13.452 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:34:13.452 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-05-08T16:34:13.964 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:34:14.966 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:34:14.966 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:34:14.966 INFO:tasks.workunit.client.0.smithi172.stdout:10 2024-05-08T16:34:14.967 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-05-08T16:34:14.967 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:34:14.967 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-05-08T16:34:15.509 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:34:16.511 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:34:16.511 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:34:16.511 INFO:tasks.workunit.client.0.smithi172.stdout:11 2024-05-08T16:34:16.512 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-05-08T16:34:16.512 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:34:16.512 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-05-08T16:34:16.947 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:34:17.949 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:34:17.949 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:34:17.950 INFO:tasks.workunit.client.0.smithi172.stdout:12 2024-05-08T16:34:17.951 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-05-08T16:34:17.951 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:34:17.951 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-05-08T16:34:18.435 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:34:19.439 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:34:19.439 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:34:19.439 INFO:tasks.workunit.client.0.smithi172.stdout:13 2024-05-08T16:34:19.440 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-05-08T16:34:19.440 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:34:19.440 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-05-08T16:34:19.937 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:34:20.939 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:34:20.939 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:34:20.939 INFO:tasks.workunit.client.0.smithi172.stdout:14 2024-05-08T16:34:20.940 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-05-08T16:34:20.940 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:34:20.940 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-05-08T16:34:21.571 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:34:22.573 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:34:22.573 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:34:22.573 INFO:tasks.workunit.client.0.smithi172.stdout:15 2024-05-08T16:34:22.574 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-05-08T16:34:22.574 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:34:22.574 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-05-08T16:34:23.491 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:34:24.493 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:34:24.494 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:34:24.494 INFO:tasks.workunit.client.0.smithi172.stdout:16 2024-05-08T16:34:24.495 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-05-08T16:34:24.495 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:34:24.495 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-05-08T16:34:25.412 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:34:26.413 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:34:26.414 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:34:26.414 INFO:tasks.workunit.client.0.smithi172.stdout:17 2024-05-08T16:34:26.414 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-05-08T16:34:26.414 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:34:26.415 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-05-08T16:34:27.207 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:34:28.209 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:34:28.209 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:34:28.209 INFO:tasks.workunit.client.0.smithi172.stdout:18 2024-05-08T16:34:28.210 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-05-08T16:34:28.210 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:34:28.210 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-05-08T16:34:29.110 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:34:30.112 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:34:30.112 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:34:30.112 INFO:tasks.workunit.client.0.smithi172.stdout:19 2024-05-08T16:34:30.113 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-05-08T16:34:30.113 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:34:30.113 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-05-08T16:34:31.048 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:34:32.049 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:34:32.049 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:34:32.049 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-05-08T16:34:32.050 INFO:tasks.workunit.client.0.smithi172.stdout:20 2024-05-08T16:34:32.050 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:34:32.050 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.8 up' 2024-05-08T16:34:32.511 INFO:tasks.workunit.client.0.smithi172.stdout:osd.8 up in weight 1 up_from 76 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6866/2736278946,v1:127.0.0.1:6867/2736278946] [v2:127.0.0.1:6868/2736278946,v1:127.0.0.1:6869/2736278946] exists,up e6c35a5a-8838-46eb-8a61-62016f175053 2024-05-08T16:34:32.512 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-08T16:34:32.512 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-08T16:34:32.512 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-08T16:34:32.512 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: get_asok_path osd.0 2024-05-08T16:34:32.512 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-05-08T16:34:32.512 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-05-08T16:34:32.513 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-08T16:34:32.513 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T16:34:32.513 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81110 2024-05-08T16:34:32.513 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.81110/ceph-osd.0.asok 2024-05-08T16:34:32.513 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: CEPH_ARGS= 2024-05-08T16:34:32.513 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: ceph --admin-daemon /tmp/ceph-asok.81110/ceph-osd.0.asok log flush 2024-05-08T16:34:32.628 INFO:tasks.workunit.client.0.smithi172.stdout:{} 2024-05-08T16:34:32.640 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:56: setup_osds: grep 'load: jerasure.*lrc' td/test-erasure-eio/osd.0.log 2024-05-08T16:34:32.650 INFO:tasks.workunit.client.0.smithi172.stdout:2024-05-08T16:28:24.418+0000 7f68a3cdf7c0 0 load: jerasure load: lrc 2024-05-08T16:34:32.652 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:423: TEST_ec_recovery_multiple_errors: local poolname=pool-jerasure 2024-05-08T16:34:32.652 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:424: TEST_ec_recovery_multiple_errors: create_erasure_coded_pool pool-jerasure 4 4 2024-05-08T16:34:32.652 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:67: create_erasure_coded_pool: local poolname=pool-jerasure 2024-05-08T16:34:32.652 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:68: create_erasure_coded_pool: shift 2024-05-08T16:34:32.652 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:69: create_erasure_coded_pool: local k=4 2024-05-08T16:34:32.652 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:70: create_erasure_coded_pool: shift 2024-05-08T16:34:32.652 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:71: create_erasure_coded_pool: local m=4 2024-05-08T16:34:32.652 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:72: create_erasure_coded_pool: shift 2024-05-08T16:34:32.652 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:74: create_erasure_coded_pool: ceph osd erasure-code-profile set myprofile plugin=jerasure k=4 m=4 crush-failure-domain=osd 2024-05-08T16:34:34.834 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:78: create_erasure_coded_pool: create_pool pool-jerasure 1 1 erasure myprofile 2024-05-08T16:34:34.834 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create pool-jerasure 1 1 erasure myprofile 2024-05-08T16:34:36.312 INFO:tasks.workunit.client.0.smithi172.stderr:pool 'pool-jerasure' already exists 2024-05-08T16:34:36.329 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-05-08T16:34:37.331 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:80: create_erasure_coded_pool: wait_for_clean 2024-05-08T16:34:37.331 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-05-08T16:34:37.331 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-05-08T16:34:37.331 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-05-08T16:34:37.332 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-05-08T16:34:37.333 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-05-08T16:34:37.333 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-05-08T16:34:37.333 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-05-08T16:34:37.333 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-05-08T16:34:37.333 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-05-08T16:34:37.458 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-05-08T16:34:37.458 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-05-08T16:34:37.458 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-05-08T16:34:37.458 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-05-08T16:34:37.458 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-05-08T16:34:37.458 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-05-08T16:34:38.037 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-05-08T16:34:38.037 INFO:tasks.workunit.client.0.smithi172.stderr:1 2024-05-08T16:34:38.038 INFO:tasks.workunit.client.0.smithi172.stderr:2 2024-05-08T16:34:38.038 INFO:tasks.workunit.client.0.smithi172.stderr:3 2024-05-08T16:34:38.038 INFO:tasks.workunit.client.0.smithi172.stderr:4 2024-05-08T16:34:38.038 INFO:tasks.workunit.client.0.smithi172.stderr:5 2024-05-08T16:34:38.038 INFO:tasks.workunit.client.0.smithi172.stderr:6 2024-05-08T16:34:38.038 INFO:tasks.workunit.client.0.smithi172.stderr:7 2024-05-08T16:34:38.038 INFO:tasks.workunit.client.0.smithi172.stderr:8' 2024-05-08T16:34:38.038 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-05-08T16:34:38.038 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T16:34:38.038 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-05-08T16:34:38.265 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=25769803852 2024-05-08T16:34:38.266 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 25769803852 2024-05-08T16:34:38.266 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803852' 2024-05-08T16:34:38.266 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T16:34:38.266 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-05-08T16:34:38.458 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=55834574917 2024-05-08T16:34:38.458 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 55834574917 2024-05-08T16:34:38.458 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803852 1-55834574917' 2024-05-08T16:34:38.458 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T16:34:38.459 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-05-08T16:34:38.616 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=81604378687 2024-05-08T16:34:38.616 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 81604378687 2024-05-08T16:34:38.616 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803852 1-55834574917 2-81604378687' 2024-05-08T16:34:38.616 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T16:34:38.617 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-05-08T16:34:38.838 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=120259084341 2024-05-08T16:34:38.838 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 120259084341 2024-05-08T16:34:38.838 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803852 1-55834574917 2-81604378687 3-120259084341' 2024-05-08T16:34:38.838 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T16:34:38.838 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-05-08T16:34:38.989 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=163208757294 2024-05-08T16:34:38.990 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 163208757294 2024-05-08T16:34:38.990 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803852 1-55834574917 2-81604378687 3-120259084341 4-163208757294' 2024-05-08T16:34:38.990 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T16:34:38.990 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-05-08T16:34:39.174 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=206158430245 2024-05-08T16:34:39.174 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 206158430245 2024-05-08T16:34:39.174 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803852 1-55834574917 2-81604378687 3-120259084341 4-163208757294 5-206158430245' 2024-05-08T16:34:39.174 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T16:34:39.174 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-05-08T16:34:39.392 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=240518168604 2024-05-08T16:34:39.392 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 240518168604 2024-05-08T16:34:39.392 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803852 1-55834574917 2-81604378687 3-120259084341 4-163208757294 5-206158430245 6-240518168604' 2024-05-08T16:34:39.392 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T16:34:39.392 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-05-08T16:34:39.538 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=283467841555 2024-05-08T16:34:39.538 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 283467841555 2024-05-08T16:34:39.538 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803852 1-55834574917 2-81604378687 3-120259084341 4-163208757294 5-206158430245 6-240518168604 7-283467841555' 2024-05-08T16:34:39.538 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T16:34:39.538 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-05-08T16:34:39.679 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=326417514505 2024-05-08T16:34:39.680 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 326417514505 2024-05-08T16:34:39.680 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803852 1-55834574917 2-81604378687 3-120259084341 4-163208757294 5-206158430245 6-240518168604 7-283467841555 8-326417514505' 2024-05-08T16:34:39.680 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-08T16:34:39.681 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-25769803852 2024-05-08T16:34:39.681 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-08T16:34:39.681 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-05-08T16:34:39.683 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-25769803852 2024-05-08T16:34:39.683 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-08T16:34:39.683 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=25769803852 2024-05-08T16:34:39.683 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 25769803852' 2024-05-08T16:34:39.684 INFO:tasks.workunit.client.0.smithi172.stdout:waiting osd.0 seq 25769803852 2024-05-08T16:34:39.684 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-05-08T16:34:40.086 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 25769803852 -lt 25769803852 2024-05-08T16:34:40.086 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-08T16:34:40.086 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-55834574917 2024-05-08T16:34:40.087 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-08T16:34:40.088 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-05-08T16:34:40.089 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-55834574917 2024-05-08T16:34:40.089 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-08T16:34:40.091 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=55834574917 2024-05-08T16:34:40.091 INFO:tasks.workunit.client.0.smithi172.stdout:waiting osd.1 seq 55834574917 2024-05-08T16:34:40.092 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 55834574917' 2024-05-08T16:34:40.092 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-05-08T16:34:40.503 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 55834574917 -lt 55834574917 2024-05-08T16:34:40.503 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-08T16:34:40.503 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-81604378687 2024-05-08T16:34:40.503 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-08T16:34:40.505 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-05-08T16:34:40.505 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-81604378687 2024-05-08T16:34:40.505 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-08T16:34:40.507 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=81604378687 2024-05-08T16:34:40.507 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 81604378687' 2024-05-08T16:34:40.507 INFO:tasks.workunit.client.0.smithi172.stdout:waiting osd.2 seq 81604378687 2024-05-08T16:34:40.507 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-05-08T16:34:40.940 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 81604378688 -lt 81604378687 2024-05-08T16:34:40.940 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-08T16:34:40.941 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-08T16:34:40.941 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-120259084341 2024-05-08T16:34:40.942 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-05-08T16:34:40.942 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-120259084341 2024-05-08T16:34:40.942 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-08T16:34:40.944 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=120259084341 2024-05-08T16:34:40.944 INFO:tasks.workunit.client.0.smithi172.stdout:waiting osd.3 seq 120259084341 2024-05-08T16:34:40.944 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 120259084341' 2024-05-08T16:34:40.944 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-05-08T16:34:41.391 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 120259084341 -lt 120259084341 2024-05-08T16:34:41.391 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-08T16:34:41.391 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-163208757294 2024-05-08T16:34:41.392 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-08T16:34:41.393 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-05-08T16:34:41.394 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-163208757294 2024-05-08T16:34:41.394 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-08T16:34:41.395 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=163208757294 2024-05-08T16:34:41.395 INFO:tasks.workunit.client.0.smithi172.stdout:waiting osd.4 seq 163208757294 2024-05-08T16:34:41.396 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 163208757294' 2024-05-08T16:34:41.396 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-05-08T16:34:41.819 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 163208757294 -lt 163208757294 2024-05-08T16:34:41.819 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-08T16:34:41.819 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-206158430245 2024-05-08T16:34:41.819 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-08T16:34:41.820 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-05-08T16:34:41.821 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-206158430245 2024-05-08T16:34:41.821 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-08T16:34:41.822 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=206158430245 2024-05-08T16:34:41.822 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 206158430245' 2024-05-08T16:34:41.822 INFO:tasks.workunit.client.0.smithi172.stdout:waiting osd.5 seq 206158430245 2024-05-08T16:34:41.823 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-05-08T16:34:42.249 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 206158430245 -lt 206158430245 2024-05-08T16:34:42.249 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-08T16:34:42.250 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-08T16:34:42.250 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-240518168604 2024-05-08T16:34:42.251 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-05-08T16:34:42.252 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-240518168604 2024-05-08T16:34:42.252 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-08T16:34:42.253 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=240518168604 2024-05-08T16:34:42.254 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 240518168604' 2024-05-08T16:34:42.254 INFO:tasks.workunit.client.0.smithi172.stdout:waiting osd.6 seq 240518168604 2024-05-08T16:34:42.254 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-05-08T16:34:42.656 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 240518168605 -lt 240518168604 2024-05-08T16:34:42.656 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-08T16:34:42.657 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 7-283467841555 2024-05-08T16:34:42.657 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-08T16:34:42.658 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=7 2024-05-08T16:34:42.659 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 7-283467841555 2024-05-08T16:34:42.659 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-08T16:34:42.660 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=283467841555 2024-05-08T16:34:42.660 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.7 seq 283467841555' 2024-05-08T16:34:42.660 INFO:tasks.workunit.client.0.smithi172.stdout:waiting osd.7 seq 283467841555 2024-05-08T16:34:42.661 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 7 2024-05-08T16:34:43.064 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 283467841555 -lt 283467841555 2024-05-08T16:34:43.064 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-08T16:34:43.065 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-326417514505 2024-05-08T16:34:43.065 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-08T16:34:43.067 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-05-08T16:34:43.067 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-326417514505 2024-05-08T16:34:43.068 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-08T16:34:43.069 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=326417514505 2024-05-08T16:34:43.069 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 326417514505' 2024-05-08T16:34:43.069 INFO:tasks.workunit.client.0.smithi172.stdout:waiting osd.8 seq 326417514505 2024-05-08T16:34:43.069 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-05-08T16:34:43.479 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 326417514505 -lt 326417514505 2024-05-08T16:34:43.480 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-05-08T16:34:43.480 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-05-08T16:34:43.480 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-05-08T16:34:44.065 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-05-08T16:34:44.065 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-05-08T16:34:44.065 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-05-08T16:34:44.066 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-05-08T16:34:44.066 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-05-08T16:34:44.066 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-05-08T16:34:44.066 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-05-08T16:34:44.066 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-05-08T16:34:44.508 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-05-08T16:34:44.509 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-05-08T16:34:44.509 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-05-08T16:34:44.509 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-05-08T16:34:45.067 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-05-08T16:34:45.067 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-05-08T16:34:45.067 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-05-08T16:34:45.067 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:426: TEST_ec_recovery_multiple_errors: rados_put td/test-erasure-eio pool-jerasure myobject 2024-05-08T16:34:45.067 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:90: rados_put: local dir=td/test-erasure-eio 2024-05-08T16:34:45.067 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:91: rados_put: local poolname=pool-jerasure 2024-05-08T16:34:45.067 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:92: rados_put: local objname=myobject 2024-05-08T16:34:45.068 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-05-08T16:34:45.068 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 AAA 2024-05-08T16:34:45.068 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-05-08T16:34:45.068 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 BBB 2024-05-08T16:34:45.068 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-05-08T16:34:45.068 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 CCCC 2024-05-08T16:34:45.068 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-05-08T16:34:45.068 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 DDDD 2024-05-08T16:34:45.068 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:100: rados_put: rados --pool pool-jerasure put myobject td/test-erasure-eio/ORIGINAL 2024-05-08T16:34:45.322 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:427: TEST_ec_recovery_multiple_errors: inject_eio ec data pool-jerasure myobject td/test-erasure-eio 0 2024-05-08T16:34:45.322 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2394: inject_eio: local pooltype=ec 2024-05-08T16:34:45.322 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2395: inject_eio: shift 2024-05-08T16:34:45.322 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2396: inject_eio: local which=data 2024-05-08T16:34:45.322 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2397: inject_eio: shift 2024-05-08T16:34:45.322 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2398: inject_eio: local poolname=pool-jerasure 2024-05-08T16:34:45.322 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2399: inject_eio: shift 2024-05-08T16:34:45.322 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2400: inject_eio: local objname=myobject 2024-05-08T16:34:45.322 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2401: inject_eio: shift 2024-05-08T16:34:45.323 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2402: inject_eio: local dir=td/test-erasure-eio 2024-05-08T16:34:45.323 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2403: inject_eio: shift 2024-05-08T16:34:45.323 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2404: inject_eio: local shard_id=0 2024-05-08T16:34:45.323 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2405: inject_eio: shift 2024-05-08T16:34:45.323 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: get_osds pool-jerasure myobject 2024-05-08T16:34:45.323 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2024-05-08T16:34:45.323 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=myobject 2024-05-08T16:34:45.323 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-05-08T16:34:45.324 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure myobject 2024-05-08T16:34:45.821 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-05-08T16:34:45.821 INFO:tasks.workunit.client.0.smithi172.stderr:5 2024-05-08T16:34:45.821 INFO:tasks.workunit.client.0.smithi172.stderr:1 2024-05-08T16:34:45.821 INFO:tasks.workunit.client.0.smithi172.stderr:0 2024-05-08T16:34:45.821 INFO:tasks.workunit.client.0.smithi172.stderr:6 2024-05-08T16:34:45.821 INFO:tasks.workunit.client.0.smithi172.stderr:2 2024-05-08T16:34:45.821 INFO:tasks.workunit.client.0.smithi172.stderr:4 2024-05-08T16:34:45.821 INFO:tasks.workunit.client.0.smithi172.stderr:7' 2024-05-08T16:34:45.821 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 5 1 0 6 2 4 7 2024-05-08T16:34:45.822 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: initial_osds=('3' '5' '1' '0' '6' '2' '4' '7') 2024-05-08T16:34:45.822 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: local -a initial_osds 2024-05-08T16:34:45.822 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2408: inject_eio: local osd_id=3 2024-05-08T16:34:45.822 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2409: inject_eio: '[' ec '!=' ec ']' 2024-05-08T16:34:45.823 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: cat td/test-erasure-eio/3/type 2024-05-08T16:34:45.823 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: type=bluestore 2024-05-08T16:34:45.823 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: set_config osd 3 bluestore_debug_inject_read_err true 2024-05-08T16:34:45.823 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-05-08T16:34:45.823 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-05-08T16:34:45.823 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-05-08T16:34:45.823 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-05-08T16:34:45.823 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-05-08T16:34:45.824 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-05-08T16:34:45.824 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-05-08T16:34:45.824 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-05-08T16:34:45.824 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-08T16:34:45.825 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T16:34:45.825 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81110 2024-05-08T16:34:45.825 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.81110/ceph-osd.3.asok 2024-05-08T16:34:45.825 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.81110/ceph-osd.3.asok config set bluestore_debug_inject_read_err true 2024-05-08T16:34:45.939 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-05-08T16:34:45.939 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local loop=0 2024-05-08T16:34:45.940 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: grep -q Invalid 2024-05-08T16:34:45.941 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: get_asok_path osd.3 2024-05-08T16:34:45.941 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-05-08T16:34:45.941 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-05-08T16:34:45.941 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-08T16:34:45.941 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T16:34:45.941 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81110 2024-05-08T16:34:45.941 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.81110/ceph-osd.3.asok 2024-05-08T16:34:45.942 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: CEPH_ARGS= 2024-05-08T16:34:45.942 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: ceph --admin-daemon /tmp/ceph-asok.81110/ceph-osd.3.asok injectdataerr pool-jerasure myobject 0 2024-05-08T16:34:46.067 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:430: TEST_ec_recovery_multiple_errors: inject_eio ec data pool-jerasure myobject td/test-erasure-eio 3 2024-05-08T16:34:46.068 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2394: inject_eio: local pooltype=ec 2024-05-08T16:34:46.068 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2395: inject_eio: shift 2024-05-08T16:34:46.068 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2396: inject_eio: local which=data 2024-05-08T16:34:46.068 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2397: inject_eio: shift 2024-05-08T16:34:46.068 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2398: inject_eio: local poolname=pool-jerasure 2024-05-08T16:34:46.068 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2399: inject_eio: shift 2024-05-08T16:34:46.068 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2400: inject_eio: local objname=myobject 2024-05-08T16:34:46.068 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2401: inject_eio: shift 2024-05-08T16:34:46.068 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2402: inject_eio: local dir=td/test-erasure-eio 2024-05-08T16:34:46.068 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2403: inject_eio: shift 2024-05-08T16:34:46.068 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2404: inject_eio: local shard_id=3 2024-05-08T16:34:46.068 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2405: inject_eio: shift 2024-05-08T16:34:46.069 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: get_osds pool-jerasure myobject 2024-05-08T16:34:46.069 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2024-05-08T16:34:46.069 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=myobject 2024-05-08T16:34:46.069 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure myobject 2024-05-08T16:34:46.069 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-05-08T16:34:46.481 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-05-08T16:34:46.481 INFO:tasks.workunit.client.0.smithi172.stderr:5 2024-05-08T16:34:46.481 INFO:tasks.workunit.client.0.smithi172.stderr:1 2024-05-08T16:34:46.481 INFO:tasks.workunit.client.0.smithi172.stderr:0 2024-05-08T16:34:46.481 INFO:tasks.workunit.client.0.smithi172.stderr:6 2024-05-08T16:34:46.481 INFO:tasks.workunit.client.0.smithi172.stderr:2 2024-05-08T16:34:46.481 INFO:tasks.workunit.client.0.smithi172.stderr:4 2024-05-08T16:34:46.481 INFO:tasks.workunit.client.0.smithi172.stderr:7' 2024-05-08T16:34:46.482 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 5 1 0 6 2 4 7 2024-05-08T16:34:46.482 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: initial_osds=('3' '5' '1' '0' '6' '2' '4' '7') 2024-05-08T16:34:46.482 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: local -a initial_osds 2024-05-08T16:34:46.482 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2408: inject_eio: local osd_id=0 2024-05-08T16:34:46.482 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2409: inject_eio: '[' ec '!=' ec ']' 2024-05-08T16:34:46.482 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: cat td/test-erasure-eio/0/type 2024-05-08T16:34:46.483 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: type=bluestore 2024-05-08T16:34:46.483 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: set_config osd 0 bluestore_debug_inject_read_err true 2024-05-08T16:34:46.483 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-05-08T16:34:46.483 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=0 2024-05-08T16:34:46.483 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-05-08T16:34:46.483 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-05-08T16:34:46.483 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-05-08T16:34:46.484 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.0 2024-05-08T16:34:46.484 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-05-08T16:34:46.484 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-05-08T16:34:46.484 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-08T16:34:46.484 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T16:34:46.484 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81110 2024-05-08T16:34:46.484 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.81110/ceph-osd.0.asok 2024-05-08T16:34:46.485 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.81110/ceph-osd.0.asok config set bluestore_debug_inject_read_err true 2024-05-08T16:34:46.602 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-05-08T16:34:46.602 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local loop=0 2024-05-08T16:34:46.603 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: grep -q Invalid 2024-05-08T16:34:46.604 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: get_asok_path osd.0 2024-05-08T16:34:46.604 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-05-08T16:34:46.604 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-05-08T16:34:46.605 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-08T16:34:46.605 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T16:34:46.605 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81110 2024-05-08T16:34:46.605 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.81110/ceph-osd.0.asok 2024-05-08T16:34:46.605 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: CEPH_ARGS= 2024-05-08T16:34:46.605 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: ceph --admin-daemon /tmp/ceph-asok.81110/ceph-osd.0.asok injectdataerr pool-jerasure myobject 3 2024-05-08T16:34:46.715 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:431: TEST_ec_recovery_multiple_errors: inject_eio ec data pool-jerasure myobject td/test-erasure-eio 4 2024-05-08T16:34:46.716 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2394: inject_eio: local pooltype=ec 2024-05-08T16:34:46.716 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2395: inject_eio: shift 2024-05-08T16:34:46.716 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2396: inject_eio: local which=data 2024-05-08T16:34:46.716 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2397: inject_eio: shift 2024-05-08T16:34:46.716 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2398: inject_eio: local poolname=pool-jerasure 2024-05-08T16:34:46.716 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2399: inject_eio: shift 2024-05-08T16:34:46.716 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2400: inject_eio: local objname=myobject 2024-05-08T16:34:46.716 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2401: inject_eio: shift 2024-05-08T16:34:46.716 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2402: inject_eio: local dir=td/test-erasure-eio 2024-05-08T16:34:46.716 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2403: inject_eio: shift 2024-05-08T16:34:46.716 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2404: inject_eio: local shard_id=4 2024-05-08T16:34:46.716 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2405: inject_eio: shift 2024-05-08T16:34:46.716 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: get_osds pool-jerasure myobject 2024-05-08T16:34:46.717 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2024-05-08T16:34:46.717 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=myobject 2024-05-08T16:34:46.717 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure myobject 2024-05-08T16:34:46.717 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-05-08T16:34:47.140 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-05-08T16:34:47.140 INFO:tasks.workunit.client.0.smithi172.stderr:5 2024-05-08T16:34:47.140 INFO:tasks.workunit.client.0.smithi172.stderr:1 2024-05-08T16:34:47.140 INFO:tasks.workunit.client.0.smithi172.stderr:0 2024-05-08T16:34:47.140 INFO:tasks.workunit.client.0.smithi172.stderr:6 2024-05-08T16:34:47.140 INFO:tasks.workunit.client.0.smithi172.stderr:2 2024-05-08T16:34:47.141 INFO:tasks.workunit.client.0.smithi172.stderr:4 2024-05-08T16:34:47.141 INFO:tasks.workunit.client.0.smithi172.stderr:7' 2024-05-08T16:34:47.141 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 5 1 0 6 2 4 7 2024-05-08T16:34:47.141 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: initial_osds=('3' '5' '1' '0' '6' '2' '4' '7') 2024-05-08T16:34:47.141 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: local -a initial_osds 2024-05-08T16:34:47.141 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2408: inject_eio: local osd_id=6 2024-05-08T16:34:47.141 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2409: inject_eio: '[' ec '!=' ec ']' 2024-05-08T16:34:47.141 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: cat td/test-erasure-eio/6/type 2024-05-08T16:34:47.142 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: type=bluestore 2024-05-08T16:34:47.142 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: set_config osd 6 bluestore_debug_inject_read_err true 2024-05-08T16:34:47.142 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-05-08T16:34:47.142 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=6 2024-05-08T16:34:47.142 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-05-08T16:34:47.142 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-05-08T16:34:47.142 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-05-08T16:34:47.143 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.6 2024-05-08T16:34:47.143 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.6 2024-05-08T16:34:47.143 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.6 ']' 2024-05-08T16:34:47.144 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-08T16:34:47.144 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T16:34:47.144 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81110 2024-05-08T16:34:47.144 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.81110/ceph-osd.6.asok 2024-05-08T16:34:47.144 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.81110/ceph-osd.6.asok config set bluestore_debug_inject_read_err true 2024-05-08T16:34:47.252 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-05-08T16:34:47.252 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local loop=0 2024-05-08T16:34:47.252 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: grep -q Invalid 2024-05-08T16:34:47.253 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: get_asok_path osd.6 2024-05-08T16:34:47.253 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.6 2024-05-08T16:34:47.253 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.6 ']' 2024-05-08T16:34:47.254 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-08T16:34:47.254 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T16:34:47.254 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81110 2024-05-08T16:34:47.254 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.81110/ceph-osd.6.asok 2024-05-08T16:34:47.255 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: CEPH_ARGS= 2024-05-08T16:34:47.255 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: ceph --admin-daemon /tmp/ceph-asok.81110/ceph-osd.6.asok injectdataerr pool-jerasure myobject 4 2024-05-08T16:34:47.376 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:433: TEST_ec_recovery_multiple_errors: get_osds pool-jerasure myobject 2024-05-08T16:34:47.377 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2024-05-08T16:34:47.377 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=myobject 2024-05-08T16:34:47.377 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure myobject 2024-05-08T16:34:47.377 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-05-08T16:34:47.796 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-05-08T16:34:47.796 INFO:tasks.workunit.client.0.smithi172.stderr:5 2024-05-08T16:34:47.796 INFO:tasks.workunit.client.0.smithi172.stderr:1 2024-05-08T16:34:47.796 INFO:tasks.workunit.client.0.smithi172.stderr:0 2024-05-08T16:34:47.796 INFO:tasks.workunit.client.0.smithi172.stderr:6 2024-05-08T16:34:47.796 INFO:tasks.workunit.client.0.smithi172.stderr:2 2024-05-08T16:34:47.796 INFO:tasks.workunit.client.0.smithi172.stderr:4 2024-05-08T16:34:47.796 INFO:tasks.workunit.client.0.smithi172.stderr:7' 2024-05-08T16:34:47.796 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 5 1 0 6 2 4 7 2024-05-08T16:34:47.796 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:433: TEST_ec_recovery_multiple_errors: initial_osds=('3' '5' '1' '0' '6' '2' '4' '7') 2024-05-08T16:34:47.796 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:433: TEST_ec_recovery_multiple_errors: local -a initial_osds 2024-05-08T16:34:47.797 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:434: TEST_ec_recovery_multiple_errors: local last_osd=7 2024-05-08T16:34:47.797 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:436: TEST_ec_recovery_multiple_errors: kill_daemons td/test-erasure-eio TERM osd.7 2024-05-08T16:34:47.797 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-08T16:34:47.797 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-08T16:34:47.797 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-08T16:34:47.797 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-08T16:34:47.797 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-08T16:34:49.107 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-08T16:34:49.108 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:437: TEST_ec_recovery_multiple_errors: ceph osd down 7 2024-05-08T16:34:49.812 INFO:tasks.workunit.client.0.smithi172.stderr:osd.7 is already down. 2024-05-08T16:34:49.831 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:438: TEST_ec_recovery_multiple_errors: ceph osd out 7 2024-05-08T16:34:50.424 INFO:tasks.workunit.client.0.smithi172.stderr:osd.7 is already out. 2024-05-08T16:34:50.440 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:441: TEST_ec_recovery_multiple_errors: wait_for_clean 2024-05-08T16:34:50.440 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-05-08T16:34:50.440 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-05-08T16:34:50.440 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-05-08T16:34:50.440 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-05-08T16:34:50.441 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-05-08T16:34:50.441 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-05-08T16:34:50.441 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-05-08T16:34:50.441 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-05-08T16:34:50.441 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-05-08T16:34:50.542 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-05-08T16:34:50.542 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-05-08T16:34:50.542 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-05-08T16:34:50.542 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-05-08T16:34:50.543 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-05-08T16:34:50.543 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-05-08T16:34:51.151 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-05-08T16:34:51.151 INFO:tasks.workunit.client.0.smithi172.stderr:1 2024-05-08T16:34:51.151 INFO:tasks.workunit.client.0.smithi172.stderr:2 2024-05-08T16:34:51.151 INFO:tasks.workunit.client.0.smithi172.stderr:3 2024-05-08T16:34:51.151 INFO:tasks.workunit.client.0.smithi172.stderr:4 2024-05-08T16:34:51.151 INFO:tasks.workunit.client.0.smithi172.stderr:5 2024-05-08T16:34:51.151 INFO:tasks.workunit.client.0.smithi172.stderr:6 2024-05-08T16:34:51.151 INFO:tasks.workunit.client.0.smithi172.stderr:7 2024-05-08T16:34:51.151 INFO:tasks.workunit.client.0.smithi172.stderr:8' 2024-05-08T16:34:51.151 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-05-08T16:34:51.151 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T16:34:51.152 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-05-08T16:34:51.330 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=25769803857 2024-05-08T16:34:51.330 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 25769803857 2024-05-08T16:34:51.330 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803857' 2024-05-08T16:34:51.330 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T16:34:51.330 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-05-08T16:34:51.680 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=55834574922 2024-05-08T16:34:51.680 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 55834574922 2024-05-08T16:34:51.680 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803857 1-55834574922' 2024-05-08T16:34:51.680 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T16:34:51.680 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-05-08T16:34:52.009 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=81604378692 2024-05-08T16:34:52.009 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 81604378692 2024-05-08T16:34:52.009 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803857 1-55834574922 2-81604378692' 2024-05-08T16:34:52.009 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T16:34:52.009 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-05-08T16:34:52.237 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=120259084346 2024-05-08T16:34:52.237 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 120259084346 2024-05-08T16:34:52.237 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803857 1-55834574922 2-81604378692 3-120259084346' 2024-05-08T16:34:52.237 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T16:34:52.237 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-05-08T16:34:52.393 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=163208757299 2024-05-08T16:34:52.393 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 163208757299 2024-05-08T16:34:52.393 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803857 1-55834574922 2-81604378692 3-120259084346 4-163208757299' 2024-05-08T16:34:52.393 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T16:34:52.393 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-05-08T16:34:52.726 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=206158430249 2024-05-08T16:34:52.726 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 206158430249 2024-05-08T16:34:52.727 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803857 1-55834574922 2-81604378692 3-120259084346 4-163208757299 5-206158430249' 2024-05-08T16:34:52.727 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T16:34:52.727 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-05-08T16:34:52.888 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=240518168609 2024-05-08T16:34:52.889 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 240518168609 2024-05-08T16:34:52.889 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803857 1-55834574922 2-81604378692 3-120259084346 4-163208757299 5-206158430249 6-240518168609' 2024-05-08T16:34:52.889 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T16:34:52.889 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2024-05-08T16:34:53.157 INFO:tasks.workunit.client.0.smithi172.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-05-08T16:34:53.160 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq= 2024-05-08T16:34:53.160 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z '' 2024-05-08T16:34:53.160 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: continue 2024-05-08T16:34:53.160 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T16:34:53.160 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2024-05-08T16:34:53.331 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=326417514510 2024-05-08T16:34:53.331 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 326417514510 2024-05-08T16:34:53.331 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803857 1-55834574922 2-81604378692 3-120259084346 4-163208757299 5-206158430249 6-240518168609 8-326417514510' 2024-05-08T16:34:53.331 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-08T16:34:53.331 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-25769803857 2024-05-08T16:34:53.332 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-08T16:34:53.332 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-05-08T16:34:53.333 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-08T16:34:53.333 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-25769803857 2024-05-08T16:34:53.334 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=25769803857 2024-05-08T16:34:53.334 INFO:tasks.workunit.client.0.smithi172.stdout:waiting osd.0 seq 25769803857 2024-05-08T16:34:53.335 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 25769803857' 2024-05-08T16:34:53.335 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-05-08T16:34:53.768 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 25769803857 -lt 25769803857 2024-05-08T16:34:53.768 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-08T16:34:53.769 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-55834574922 2024-05-08T16:34:53.769 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-08T16:34:53.770 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-05-08T16:34:53.771 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-55834574922 2024-05-08T16:34:53.771 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-08T16:34:53.772 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=55834574922 2024-05-08T16:34:53.772 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 55834574922' 2024-05-08T16:34:53.772 INFO:tasks.workunit.client.0.smithi172.stdout:waiting osd.1 seq 55834574922 2024-05-08T16:34:53.772 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-05-08T16:34:54.367 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 55834574922 -lt 55834574922 2024-05-08T16:34:54.367 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-08T16:34:54.368 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-81604378692 2024-05-08T16:34:54.368 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-08T16:34:54.369 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-05-08T16:34:54.370 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-81604378692 2024-05-08T16:34:54.370 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-08T16:34:54.371 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=81604378692 2024-05-08T16:34:54.371 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 81604378692' 2024-05-08T16:34:54.372 INFO:tasks.workunit.client.0.smithi172.stdout:waiting osd.2 seq 81604378692 2024-05-08T16:34:54.372 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-05-08T16:34:54.814 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 81604378692 -lt 81604378692 2024-05-08T16:34:54.814 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-08T16:34:54.815 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-120259084346 2024-05-08T16:34:54.815 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-08T16:34:54.816 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-05-08T16:34:54.817 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-120259084346 2024-05-08T16:34:54.817 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-08T16:34:54.818 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=120259084346 2024-05-08T16:34:54.819 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 120259084346' 2024-05-08T16:34:54.819 INFO:tasks.workunit.client.0.smithi172.stdout:waiting osd.3 seq 120259084346 2024-05-08T16:34:54.819 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-05-08T16:34:55.247 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 120259084346 -lt 120259084346 2024-05-08T16:34:55.247 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-08T16:34:55.248 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-163208757299 2024-05-08T16:34:55.248 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-08T16:34:55.249 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-05-08T16:34:55.250 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-163208757299 2024-05-08T16:34:55.250 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-08T16:34:55.251 INFO:tasks.workunit.client.0.smithi172.stdout:waiting osd.4 seq 163208757299 2024-05-08T16:34:55.251 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=163208757299 2024-05-08T16:34:55.251 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 163208757299' 2024-05-08T16:34:55.252 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-05-08T16:34:55.676 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 163208757299 -lt 163208757299 2024-05-08T16:34:55.676 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-08T16:34:55.677 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-206158430249 2024-05-08T16:34:55.677 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-08T16:34:55.679 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-05-08T16:34:55.680 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-206158430249 2024-05-08T16:34:55.680 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-08T16:34:55.681 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=206158430249 2024-05-08T16:34:55.681 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 206158430249' 2024-05-08T16:34:55.682 INFO:tasks.workunit.client.0.smithi172.stdout:waiting osd.5 seq 206158430249 2024-05-08T16:34:55.682 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-05-08T16:34:56.102 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 206158430250 -lt 206158430249 2024-05-08T16:34:56.102 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-08T16:34:56.103 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-08T16:34:56.103 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-240518168609 2024-05-08T16:34:56.104 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-05-08T16:34:56.105 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-240518168609 2024-05-08T16:34:56.105 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-08T16:34:56.106 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=240518168609 2024-05-08T16:34:56.106 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 240518168609' 2024-05-08T16:34:56.106 INFO:tasks.workunit.client.0.smithi172.stdout:waiting osd.6 seq 240518168609 2024-05-08T16:34:56.106 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-05-08T16:34:56.532 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 240518168609 -lt 240518168609 2024-05-08T16:34:56.532 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-08T16:34:56.533 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 8-326417514510 2024-05-08T16:34:56.533 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-08T16:34:56.534 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=8 2024-05-08T16:34:56.535 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-08T16:34:56.535 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 8-326417514510 2024-05-08T16:34:56.536 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=326417514510 2024-05-08T16:34:56.537 INFO:tasks.workunit.client.0.smithi172.stdout:waiting osd.8 seq 326417514510 2024-05-08T16:34:56.537 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.8 seq 326417514510' 2024-05-08T16:34:56.537 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 8 2024-05-08T16:34:56.970 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 326417514510 -lt 326417514510 2024-05-08T16:34:56.970 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-05-08T16:34:56.971 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-05-08T16:34:56.971 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-05-08T16:34:57.544 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-05-08T16:34:57.544 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-05-08T16:34:57.545 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-05-08T16:34:57.546 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-05-08T16:34:57.546 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-05-08T16:34:57.546 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-05-08T16:34:57.546 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-05-08T16:34:57.546 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-05-08T16:34:57.971 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-05-08T16:34:57.971 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-05-08T16:34:57.972 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-05-08T16:34:57.972 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-05-08T16:34:58.499 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 5 2024-05-08T16:34:58.499 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' -1 2024-05-08T16:34:58.499 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-05-08T16:34:58.499 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=4 2024-05-08T16:34:58.499 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-05-08T16:34:58.500 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-05-08T16:34:58.600 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-05-08T16:34:58.601 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-05-08T16:34:58.602 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-05-08T16:34:58.602 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-05-08T16:34:58.602 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-05-08T16:34:58.602 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-05-08T16:34:58.602 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-05-08T16:34:58.602 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-05-08T16:34:59.019 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-05-08T16:34:59.020 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-05-08T16:34:59.020 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-05-08T16:34:59.020 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-05-08T16:34:59.545 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-05-08T16:34:59.545 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-05-08T16:34:59.545 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-05-08T16:34:59.545 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:443: TEST_ec_recovery_multiple_errors: rados_get td/test-erasure-eio pool-jerasure myobject 2024-05-08T16:34:59.545 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2024-05-08T16:34:59.545 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2024-05-08T16:34:59.545 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=myobject 2024-05-08T16:34:59.545 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=ok 2024-05-08T16:34:59.545 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' ok = fail ']' 2024-05-08T16:34:59.546 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:120: rados_get: rados --pool pool-jerasure get myobject td/test-erasure-eio/COPY 2024-05-08T16:34:59.587 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:121: rados_get: diff td/test-erasure-eio/ORIGINAL td/test-erasure-eio/COPY 2024-05-08T16:34:59.588 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:122: rados_get: rm td/test-erasure-eio/COPY 2024-05-08T16:34:59.589 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:445: TEST_ec_recovery_multiple_errors: delete_erasure_coded_pool pool-jerasure 2024-05-08T16:34:59.589 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:84: delete_erasure_coded_pool: local poolname=pool-jerasure 2024-05-08T16:34:59.589 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:85: delete_erasure_coded_pool: ceph osd pool delete pool-jerasure pool-jerasure --yes-i-really-really-mean-it 2024-05-08T16:35:00.082 INFO:tasks.workunit.client.0.smithi172.stderr:pool 'pool-jerasure' does not exist 2024-05-08T16:35:00.098 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:86: delete_erasure_coded_pool: ceph osd erasure-code-profile rm myprofile 2024-05-08T16:35:01.006 INFO:tasks.workunit.client.0.smithi172.stderr:erasure-code-profile myprofile does not exist 2024-05-08T16:35:01.024 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:42: run: teardown td/test-erasure-eio 2024-05-08T16:35:01.024 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-eio 2024-05-08T16:35:01.024 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-05-08T16:35:01.024 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-eio KILL 2024-05-08T16:35:01.024 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-08T16:35:01.024 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-08T16:35:01.025 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-08T16:35:01.025 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-08T16:35:01.025 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-08T16:35:01.340 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-08T16:35:01.341 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-05-08T16:35:01.342 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-05-08T16:35:01.342 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-05-08T16:35:01.343 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-05-08T16:35:01.343 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-05-08T16:35:01.344 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-05-08T16:35:01.345 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-08T16:35:01.345 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-05-08T16:35:01.345 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-05-08T16:35:01.346 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-08T16:35:01.347 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-05-08T16:35:01.348 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-05-08T16:35:01.348 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-eio 2024-05-08T16:35:01.402 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-05-08T16:35:01.402 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T16:35:01.402 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81110 2024-05-08T16:35:01.403 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.81110 2024-05-08T16:35:01.404 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-05-08T16:35:01.405 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-05-08T16:35:01.405 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:32: run: for func in $funcs 2024-05-08T16:35:01.405 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:33: run: setup td/test-erasure-eio 2024-05-08T16:35:01.405 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/test-erasure-eio 2024-05-08T16:35:01.405 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/test-erasure-eio 2024-05-08T16:35:01.405 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-eio 2024-05-08T16:35:01.405 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-05-08T16:35:01.405 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-eio KILL 2024-05-08T16:35:01.405 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-08T16:35:01.405 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-08T16:35:01.406 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-08T16:35:01.406 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-08T16:35:01.406 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-08T16:35:01.406 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-08T16:35:01.407 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-05-08T16:35:01.408 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-05-08T16:35:01.408 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-05-08T16:35:01.409 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-05-08T16:35:01.410 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-05-08T16:35:01.410 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-05-08T16:35:01.411 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-08T16:35:01.411 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-05-08T16:35:01.412 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-05-08T16:35:01.413 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-08T16:35:01.414 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-05-08T16:35:01.414 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-05-08T16:35:01.414 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-eio 2024-05-08T16:35:01.415 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-05-08T16:35:01.415 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T16:35:01.417 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81110 2024-05-08T16:35:01.417 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.81110 2024-05-08T16:35:01.418 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-05-08T16:35:01.418 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-05-08T16:35:01.418 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/test-erasure-eio 2024-05-08T16:35:01.418 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-05-08T16:35:01.418 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T16:35:01.419 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81110 2024-05-08T16:35:01.419 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.81110 2024-05-08T16:35:01.420 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-05-08T16:35:01.420 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-05-08T16:35:01.420 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-05-08T16:35:01.420 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/test-erasure-eio 1' TERM HUP INT 2024-05-08T16:35:01.420 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:34: run: run_mon td/test-erasure-eio a 2024-05-08T16:35:01.420 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/test-erasure-eio 2024-05-08T16:35:01.420 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-05-08T16:35:01.420 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-05-08T16:35:01.420 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-05-08T16:35:01.420 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/test-erasure-eio/a 2024-05-08T16:35:01.420 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-eio/a --run-dir=td/test-erasure-eio 2024-05-08T16:35:01.841 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-05-08T16:35:01.841 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-08T16:35:01.841 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-08T16:35:01.842 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-08T16:35:01.842 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T16:35:01.842 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81110 2024-05-08T16:35:01.842 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.81110/$cluster-$name.asok' 2024-05-08T16:35:01.842 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-eio/a '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.81110/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-eio/log --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-05-08T16:35:02.082 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-05-08T16:35:02.084 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-05-08T16:35:02.084 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-05-08T16:35:02.084 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-05-08T16:35:02.084 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-05-08T16:35:02.084 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-05-08T16:35:02.085 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-05-08T16:35:02.085 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-05-08T16:35:02.085 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-05-08T16:35:02.085 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-08T16:35:02.085 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T16:35:02.086 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81110 2024-05-08T16:35:02.086 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.81110/ceph-mon.a.asok 2024-05-08T16:35:02.086 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-05-08T16:35:02.086 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.81110/ceph-mon.a.asok config get fsid 2024-05-08T16:35:02.204 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-05-08T16:35:02.204 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-05-08T16:35:02.204 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-05-08T16:35:02.205 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-05-08T16:35:02.205 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-05-08T16:35:02.206 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-05-08T16:35:02.206 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-05-08T16:35:02.206 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-05-08T16:35:02.206 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-08T16:35:02.206 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T16:35:02.206 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81110 2024-05-08T16:35:02.206 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.81110/ceph-mon.a.asok 2024-05-08T16:35:02.207 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-05-08T16:35:02.207 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.81110/ceph-mon.a.asok config get mon_host 2024-05-08T16:35:02.308 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:35: run: run_mgr td/test-erasure-eio x 2024-05-08T16:35:02.308 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/test-erasure-eio 2024-05-08T16:35:02.308 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-05-08T16:35:02.308 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-05-08T16:35:02.308 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-05-08T16:35:02.309 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/test-erasure-eio/x 2024-05-08T16:35:02.309 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-05-08T16:35:03.206 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-05-08T16:35:03.206 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-08T16:35:03.206 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-08T16:35:03.207 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-08T16:35:03.207 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T16:35:03.207 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81110 2024-05-08T16:35:03.207 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.81110/$cluster-$name.asok' 2024-05-08T16:35:03.208 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-05-08T16:35:03.209 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-eio/x '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.81110/$cluster-$name.asok' --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-05-08T16:35:03.241 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:36: run: create_pool rbd 4 2024-05-08T16:35:03.241 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-05-08T16:35:03.580 INFO:tasks.workunit.client.0.smithi172.stderr:pool 'rbd' already exists 2024-05-08T16:35:03.598 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-05-08T16:35:04.600 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: get_asok_path mon.a 2024-05-08T16:35:04.600 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-05-08T16:35:04.601 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-05-08T16:35:04.601 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-08T16:35:04.601 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T16:35:04.601 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81110 2024-05-08T16:35:04.601 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.81110/ceph-mon.a.asok 2024-05-08T16:35:04.602 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: CEPH_ARGS= 2024-05-08T16:35:04.602 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: ceph --admin-daemon /tmp/ceph-asok.81110/ceph-mon.a.asok log flush 2024-05-08T16:35:04.703 INFO:tasks.workunit.client.0.smithi172.stdout:{} 2024-05-08T16:35:04.717 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:40: run: grep 'load: jerasure.*lrc' td/test-erasure-eio/mon.a.log 2024-05-08T16:35:04.720 INFO:tasks.workunit.client.0.smithi172.stdout:2024-05-08T16:35:01.888+0000 7f4c70d18c80 0 load: jerasure load: lrc 2024-05-08T16:35:04.721 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:41: run: TEST_ec_recovery_multiple_objects td/test-erasure-eio 2024-05-08T16:35:04.721 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:451: TEST_ec_recovery_multiple_objects: local dir=td/test-erasure-eio 2024-05-08T16:35:04.721 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:452: TEST_ec_recovery_multiple_objects: local objname=myobject 2024-05-08T16:35:04.721 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:454: TEST_ec_recovery_multiple_objects: ORIG_ARGS='--fsid=4457b0f1-028b-482b-8ee3-7ee05861e6cc --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-05-08T16:35:04.721 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:455: TEST_ec_recovery_multiple_objects: CEPH_ARGS+=' --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 ' 2024-05-08T16:35:04.722 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:456: TEST_ec_recovery_multiple_objects: setup_osds 7 2024-05-08T16:35:04.722 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:47: setup_osds: local count=7 2024-05-08T16:35:04.722 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:48: setup_osds: shift 2024-05-08T16:35:04.722 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: expr 7 - 1 2024-05-08T16:35:04.723 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: seq 0 6 2024-05-08T16:35:04.726 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-05-08T16:35:04.726 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 0 2024-05-08T16:35:04.726 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-05-08T16:35:04.726 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-05-08T16:35:04.726 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-05-08T16:35:04.726 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-05-08T16:35:04.726 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/0 2024-05-08T16:35:04.726 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=4457b0f1-028b-482b-8ee3-7ee05861e6cc --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 ' 2024-05-08T16:35:04.727 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-08T16:35:04.727 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-05-08T16:35:04.727 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-08T16:35:04.727 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/0' 2024-05-08T16:35:04.727 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/0/journal' 2024-05-08T16:35:04.727 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-05-08T16:35:04.727 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-05-08T16:35:04.727 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-05-08T16:35:04.727 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-05-08T16:35:04.727 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-08T16:35:04.727 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-08T16:35:04.727 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-08T16:35:04.727 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T16:35:04.727 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81110 2024-05-08T16:35:04.728 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.81110/$cluster-$name.asok' 2024-05-08T16:35:04.728 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.81110/$cluster-$name.asok' 2024-05-08T16:35:04.728 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-05-08T16:35:04.728 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-05-08T16:35:04.728 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-05-08T16:35:04.728 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-05-08T16:35:04.728 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-05-08T16:35:04.728 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-08T16:35:04.728 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-08T16:35:04.728 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-08T16:35:04.728 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-08T16:35:04.728 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-05-08T16:35:04.728 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-05-08T16:35:04.728 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/0 2024-05-08T16:35:04.729 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-05-08T16:35:04.729 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=09883b93-75d7-49cb-8b1e-501975993158 2024-05-08T16:35:04.729 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 09883b93-75d7-49cb-8b1e-501975993158' 2024-05-08T16:35:04.729 INFO:tasks.workunit.client.0.smithi172.stdout:add osd0 09883b93-75d7-49cb-8b1e-501975993158 2024-05-08T16:35:04.730 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-05-08T16:35:04.755 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQC4qTtmCz9OLRAAl7S1SqJwRA+W6BE3OCnckw== 2024-05-08T16:35:04.755 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQC4qTtmCz9OLRAAl7S1SqJwRA+W6BE3OCnckw=="}' 2024-05-08T16:35:04.755 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 09883b93-75d7-49cb-8b1e-501975993158 -i td/test-erasure-eio/0/new.json 2024-05-08T16:35:05.080 INFO:tasks.workunit.client.0.smithi172.stdout:0 2024-05-08T16:35:05.097 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/0/new.json 2024-05-08T16:35:05.098 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=4457b0f1-028b-482b-8ee3-7ee05861e6cc --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81110/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQC4qTtmCz9OLRAAl7S1SqJwRA+W6BE3OCnckw== --osd-uuid 09883b93-75d7-49cb-8b1e-501975993158 2024-05-08T16:35:05.131 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:35:05.132+0000 7fb97f64b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:35:05.157 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:35:05.160+0000 7fb97f64b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:35:05.159 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:35:05.160+0000 7fb97f64b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:35:05.159 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:35:05.160+0000 7fb97f64b7c0 -1 bluestore(td/test-erasure-eio/0/block) _read_bdev_label failed to open td/test-erasure-eio/0/block: (2) No such file or directory 2024-05-08T16:35:05.159 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:35:05.160+0000 7fb97f64b7c0 -1 bluestore(td/test-erasure-eio/0/block) _read_bdev_label failed to open td/test-erasure-eio/0/block: (2) No such file or directory 2024-05-08T16:35:05.159 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:35:05.160+0000 7fb97f64b7c0 -1 bluestore(td/test-erasure-eio/0/block) _read_bdev_label failed to open td/test-erasure-eio/0/block: (2) No such file or directory 2024-05-08T16:35:05.197 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:35:05.200+0000 7fb97f64b7c0 -1 bluestore(td/test-erasure-eio/0) _read_fsid unparsable uuid 2024-05-08T16:35:12.418 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/0/keyring 2024-05-08T16:35:12.418 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-05-08T16:35:12.420 INFO:tasks.workunit.client.0.smithi172.stdout:adding osd0 key to auth repository 2024-05-08T16:35:12.420 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-05-08T16:35:12.421 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-05-08T16:35:12.958 INFO:tasks.workunit.client.0.smithi172.stdout:start osd.0 2024-05-08T16:35:12.959 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-05-08T16:35:12.960 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=4457b0f1-028b-482b-8ee3-7ee05861e6cc --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81110/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-05-08T16:35:12.960 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-05-08T16:35:12.960 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-05-08T16:35:12.960 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-05-08T16:35:12.993 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:35:12.996+0000 7f31515817c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:35:12.995 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:35:12.996+0000 7f31515817c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:35:12.997 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:35:13.000+0000 7f31515817c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:35:13.344 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-05-08T16:35:13.344 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-08T16:35:13.344 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-05-08T16:35:13.344 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-08T16:35:13.345 INFO:tasks.workunit.client.0.smithi172.stdout:0 2024-05-08T16:35:13.345 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-08T16:35:13.345 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:35:13.345 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-08T16:35:13.345 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:35:13.345 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:35:13.566 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:35:13.568+0000 7f31515817c0 -1 Falling back to public interface 2024-05-08T16:35:13.782 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:35:14.784 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:35:14.784 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:35:14.784 INFO:tasks.workunit.client.0.smithi172.stdout:1 2024-05-08T16:35:14.784 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-08T16:35:14.785 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:35:14.785 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:35:15.192 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:35:16.194 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:35:16.194 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:35:16.194 INFO:tasks.workunit.client.0.smithi172.stdout:2 2024-05-08T16:35:16.195 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-08T16:35:16.195 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:35:16.195 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:35:16.195 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:35:16.196+0000 7f31515817c0 -1 osd.0 0 log_to_monitors true 2024-05-08T16:35:16.717 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:35:17.718 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:35:17.719 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:35:17.719 INFO:tasks.workunit.client.0.smithi172.stdout:3 2024-05-08T16:35:17.719 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-05-08T16:35:17.719 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:35:17.720 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:35:18.237 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:35:19.239 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:35:19.239 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:35:19.240 INFO:tasks.workunit.client.0.smithi172.stdout:4 2024-05-08T16:35:19.240 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-05-08T16:35:19.240 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:35:19.240 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:35:19.728 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:35:20.730 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:35:20.730 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:35:20.730 INFO:tasks.workunit.client.0.smithi172.stdout:5 2024-05-08T16:35:20.731 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-05-08T16:35:20.731 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:35:20.731 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:35:21.206 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:35:22.207 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:35:22.207 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:35:22.208 INFO:tasks.workunit.client.0.smithi172.stdout:6 2024-05-08T16:35:22.208 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-05-08T16:35:22.208 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:35:22.208 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:35:22.669 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:35:23.671 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:35:23.671 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:35:23.671 INFO:tasks.workunit.client.0.smithi172.stdout:7 2024-05-08T16:35:23.672 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-05-08T16:35:23.672 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:35:23.672 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:35:24.260 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:35:25.262 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:35:25.262 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:35:25.262 INFO:tasks.workunit.client.0.smithi172.stdout:8 2024-05-08T16:35:25.263 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-05-08T16:35:25.263 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:35:25.263 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:35:25.722 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:35:26.723 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:35:26.724 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:35:26.724 INFO:tasks.workunit.client.0.smithi172.stdout:9 2024-05-08T16:35:26.724 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-05-08T16:35:26.725 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:35:26.725 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:35:27.204 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:35:28.205 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:35:28.206 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:35:28.206 INFO:tasks.workunit.client.0.smithi172.stdout:10 2024-05-08T16:35:28.207 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-05-08T16:35:28.207 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:35:28.207 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:35:28.805 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:35:29.810 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:35:29.810 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:35:29.810 INFO:tasks.workunit.client.0.smithi172.stdout:11 2024-05-08T16:35:29.811 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-05-08T16:35:29.811 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:35:29.811 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:35:30.346 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:35:31.348 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:35:31.348 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:35:31.348 INFO:tasks.workunit.client.0.smithi172.stdout:12 2024-05-08T16:35:31.348 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-05-08T16:35:31.349 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:35:31.349 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:35:32.189 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:35:33.190 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:35:33.191 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:35:33.191 INFO:tasks.workunit.client.0.smithi172.stdout:13 2024-05-08T16:35:33.191 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-05-08T16:35:33.192 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:35:33.192 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:35:34.109 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:35:35.110 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:35:35.111 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:35:35.111 INFO:tasks.workunit.client.0.smithi172.stdout:14 2024-05-08T16:35:35.111 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-05-08T16:35:35.111 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:35:35.111 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:35:36.109 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:35:37.111 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:35:37.111 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:35:37.112 INFO:tasks.workunit.client.0.smithi172.stdout:15 2024-05-08T16:35:37.112 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-05-08T16:35:37.112 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:35:37.112 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:35:37.875 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:35:38.876 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:35:38.877 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:35:38.877 INFO:tasks.workunit.client.0.smithi172.stdout:16 2024-05-08T16:35:38.877 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-05-08T16:35:38.878 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:35:38.878 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:35:39.559 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:35:40.560 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:35:40.560 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:35:40.561 INFO:tasks.workunit.client.0.smithi172.stdout:17 2024-05-08T16:35:40.561 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-05-08T16:35:40.561 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:35:40.561 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:35:41.493 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:35:42.495 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:35:42.495 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:35:42.495 INFO:tasks.workunit.client.0.smithi172.stdout:18 2024-05-08T16:35:42.496 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-05-08T16:35:42.496 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:35:42.496 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:35:42.984 INFO:tasks.workunit.client.0.smithi172.stdout:osd.0 up in weight 1 up_from 6 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/553317705,v1:127.0.0.1:6803/553317705] [v2:127.0.0.1:6804/553317705,v1:127.0.0.1:6805/553317705] exists,up 09883b93-75d7-49cb-8b1e-501975993158 2024-05-08T16:35:42.988 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-08T16:35:42.988 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-08T16:35:42.988 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-08T16:35:42.988 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-05-08T16:35:42.988 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 1 2024-05-08T16:35:42.988 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-05-08T16:35:42.988 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-05-08T16:35:42.988 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-05-08T16:35:42.989 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-05-08T16:35:42.989 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/1 2024-05-08T16:35:42.989 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=4457b0f1-028b-482b-8ee3-7ee05861e6cc --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 ' 2024-05-08T16:35:42.989 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-08T16:35:42.989 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-05-08T16:35:42.989 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-08T16:35:42.989 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/1' 2024-05-08T16:35:42.989 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/1/journal' 2024-05-08T16:35:42.989 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-05-08T16:35:42.989 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-05-08T16:35:42.989 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-05-08T16:35:42.989 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-05-08T16:35:42.989 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-08T16:35:42.989 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-08T16:35:42.989 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-08T16:35:42.990 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T16:35:42.990 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81110 2024-05-08T16:35:42.990 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.81110/$cluster-$name.asok' 2024-05-08T16:35:42.990 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.81110/$cluster-$name.asok' 2024-05-08T16:35:42.990 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-05-08T16:35:42.990 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-05-08T16:35:42.990 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-05-08T16:35:42.990 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-05-08T16:35:42.990 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-05-08T16:35:42.990 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-08T16:35:42.990 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-08T16:35:42.990 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-08T16:35:42.990 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-08T16:35:42.991 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-05-08T16:35:42.991 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-05-08T16:35:42.991 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/1 2024-05-08T16:35:42.991 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-05-08T16:35:42.992 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=242106bc-77dc-49d2-9c89-67134ace03f7 2024-05-08T16:35:42.992 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 242106bc-77dc-49d2-9c89-67134ace03f7' 2024-05-08T16:35:42.992 INFO:tasks.workunit.client.0.smithi172.stdout:add osd1 242106bc-77dc-49d2-9c89-67134ace03f7 2024-05-08T16:35:42.992 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-05-08T16:35:43.015 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDfqTtmMQM2ARAAdfMx/rDN2DwER3T7ms67Rg== 2024-05-08T16:35:43.015 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDfqTtmMQM2ARAAdfMx/rDN2DwER3T7ms67Rg=="}' 2024-05-08T16:35:43.015 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 242106bc-77dc-49d2-9c89-67134ace03f7 -i td/test-erasure-eio/1/new.json 2024-05-08T16:35:44.038 INFO:tasks.workunit.client.0.smithi172.stdout:1 2024-05-08T16:35:44.061 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/1/new.json 2024-05-08T16:35:44.062 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=4457b0f1-028b-482b-8ee3-7ee05861e6cc --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81110/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDfqTtmMQM2ARAAdfMx/rDN2DwER3T7ms67Rg== --osd-uuid 242106bc-77dc-49d2-9c89-67134ace03f7 2024-05-08T16:35:44.100 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:35:44.103+0000 7f1085ad57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:35:44.105 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:35:44.107+0000 7f1085ad57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:35:44.106 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:35:44.111+0000 7f1085ad57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:35:44.107 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:35:44.111+0000 7f1085ad57c0 -1 bluestore(td/test-erasure-eio/1/block) _read_bdev_label failed to open td/test-erasure-eio/1/block: (2) No such file or directory 2024-05-08T16:35:44.107 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:35:44.111+0000 7f1085ad57c0 -1 bluestore(td/test-erasure-eio/1/block) _read_bdev_label failed to open td/test-erasure-eio/1/block: (2) No such file or directory 2024-05-08T16:35:44.107 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:35:44.111+0000 7f1085ad57c0 -1 bluestore(td/test-erasure-eio/1/block) _read_bdev_label failed to open td/test-erasure-eio/1/block: (2) No such file or directory 2024-05-08T16:35:44.235 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:35:44.239+0000 7f1085ad57c0 -1 bluestore(td/test-erasure-eio/1) _read_fsid unparsable uuid 2024-05-08T16:35:51.135 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/1/keyring 2024-05-08T16:35:51.135 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-05-08T16:35:51.136 INFO:tasks.workunit.client.0.smithi172.stdout:adding osd1 key to auth repository 2024-05-08T16:35:51.136 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-05-08T16:35:51.137 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-05-08T16:35:51.713 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-05-08T16:35:51.714 INFO:tasks.workunit.client.0.smithi172.stdout:start osd.1 2024-05-08T16:35:51.715 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-05-08T16:35:51.715 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-05-08T16:35:51.715 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=4457b0f1-028b-482b-8ee3-7ee05861e6cc --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81110/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-05-08T16:35:51.715 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-05-08T16:35:51.751 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:35:51.755+0000 7f2b84c0c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:35:51.795 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:35:51.799+0000 7f2b84c0c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:35:51.796 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:35:51.799+0000 7f2b84c0c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:35:52.168 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-05-08T16:35:52.169 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-08T16:35:52.169 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-05-08T16:35:52.169 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-08T16:35:52.169 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-08T16:35:52.169 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:35:52.169 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-08T16:35:52.170 INFO:tasks.workunit.client.0.smithi172.stdout:0 2024-05-08T16:35:52.170 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:35:52.170 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T16:35:52.360 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:35:52.363+0000 7f2b84c0c7c0 -1 Falling back to public interface 2024-05-08T16:35:52.596 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:35:53.597 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:35:53.597 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:35:53.598 INFO:tasks.workunit.client.0.smithi172.stdout:1 2024-05-08T16:35:53.598 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-08T16:35:53.598 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:35:53.598 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T16:35:54.018 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:35:55.019 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:35:55.020 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:35:55.020 INFO:tasks.workunit.client.0.smithi172.stdout:2 2024-05-08T16:35:55.021 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-08T16:35:55.021 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:35:55.021 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T16:35:55.047 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:35:55.050+0000 7f2b84c0c7c0 -1 osd.1 0 log_to_monitors true 2024-05-08T16:35:55.504 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:35:56.506 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:35:56.506 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:35:56.506 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-05-08T16:35:56.507 INFO:tasks.workunit.client.0.smithi172.stdout:3 2024-05-08T16:35:56.507 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:35:56.507 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T16:35:56.974 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:35:57.976 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:35:57.976 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:35:57.976 INFO:tasks.workunit.client.0.smithi172.stdout:4 2024-05-08T16:35:57.977 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-05-08T16:35:57.977 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:35:57.977 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T16:35:58.630 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:35:59.632 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:35:59.632 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:35:59.632 INFO:tasks.workunit.client.0.smithi172.stdout:5 2024-05-08T16:35:59.633 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-05-08T16:35:59.633 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:35:59.633 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T16:36:00.365 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:36:01.367 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:36:01.367 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:36:01.368 INFO:tasks.workunit.client.0.smithi172.stdout:6 2024-05-08T16:36:01.368 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-05-08T16:36:01.368 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:36:01.368 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T16:36:01.859 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:36:02.862 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:36:02.862 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:36:02.862 INFO:tasks.workunit.client.0.smithi172.stdout:7 2024-05-08T16:36:02.863 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-05-08T16:36:02.863 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:36:02.863 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T16:36:03.360 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:36:04.361 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:36:04.361 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:36:04.361 INFO:tasks.workunit.client.0.smithi172.stdout:8 2024-05-08T16:36:04.362 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-05-08T16:36:04.362 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:36:04.362 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T16:36:05.074 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:36:06.076 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:36:06.077 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:36:06.077 INFO:tasks.workunit.client.0.smithi172.stdout:9 2024-05-08T16:36:06.077 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-05-08T16:36:06.077 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:36:06.078 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T16:36:06.648 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:36:07.649 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:36:07.649 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:36:07.650 INFO:tasks.workunit.client.0.smithi172.stdout:10 2024-05-08T16:36:07.650 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-05-08T16:36:07.650 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:36:07.650 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T16:36:08.299 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:36:09.301 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:36:09.301 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:36:09.301 INFO:tasks.workunit.client.0.smithi172.stdout:11 2024-05-08T16:36:09.301 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-05-08T16:36:09.302 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:36:09.302 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T16:36:09.769 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:36:10.771 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:36:10.771 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:36:10.772 INFO:tasks.workunit.client.0.smithi172.stdout:12 2024-05-08T16:36:10.773 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-05-08T16:36:10.773 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:36:10.773 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T16:36:12.016 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:36:13.017 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:36:13.017 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:36:13.018 INFO:tasks.workunit.client.0.smithi172.stdout:13 2024-05-08T16:36:13.018 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-05-08T16:36:13.019 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:36:13.019 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T16:36:13.843 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:36:14.844 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:36:14.845 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:36:14.845 INFO:tasks.workunit.client.0.smithi172.stdout:14 2024-05-08T16:36:14.845 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-05-08T16:36:14.845 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:36:14.846 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T16:36:15.757 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:36:16.759 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:36:16.759 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:36:16.760 INFO:tasks.workunit.client.0.smithi172.stdout:15 2024-05-08T16:36:16.760 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-05-08T16:36:16.760 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:36:16.760 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T16:36:17.452 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:36:18.453 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:36:18.453 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:36:18.454 INFO:tasks.workunit.client.0.smithi172.stdout:16 2024-05-08T16:36:18.454 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-05-08T16:36:18.454 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:36:18.454 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T16:36:19.325 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:36:20.327 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:36:20.327 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:36:20.327 INFO:tasks.workunit.client.0.smithi172.stdout:17 2024-05-08T16:36:20.328 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-05-08T16:36:20.328 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:36:20.328 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T16:36:21.460 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:36:22.462 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:36:22.463 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:36:22.463 INFO:tasks.workunit.client.0.smithi172.stdout:18 2024-05-08T16:36:22.464 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-05-08T16:36:22.464 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:36:22.464 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T16:36:22.958 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:36:23.961 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:36:23.961 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:36:23.961 INFO:tasks.workunit.client.0.smithi172.stdout:19 2024-05-08T16:36:23.962 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-05-08T16:36:23.962 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:36:23.962 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T16:36:24.511 INFO:tasks.workunit.client.0.smithi172.stdout:osd.1 up in weight 1 up_from 13 up_thru 13 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/3086508659,v1:127.0.0.1:6811/3086508659] [v2:127.0.0.1:6812/3086508659,v1:127.0.0.1:6813/3086508659] exists,up 242106bc-77dc-49d2-9c89-67134ace03f7 2024-05-08T16:36:24.516 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-08T16:36:24.516 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-08T16:36:24.516 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-08T16:36:24.516 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-05-08T16:36:24.517 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 2 2024-05-08T16:36:24.517 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-05-08T16:36:24.517 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-05-08T16:36:24.517 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-05-08T16:36:24.517 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-05-08T16:36:24.517 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/2 2024-05-08T16:36:24.517 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=4457b0f1-028b-482b-8ee3-7ee05861e6cc --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 ' 2024-05-08T16:36:24.517 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-08T16:36:24.517 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-05-08T16:36:24.517 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-08T16:36:24.517 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/2' 2024-05-08T16:36:24.517 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/2/journal' 2024-05-08T16:36:24.517 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-05-08T16:36:24.517 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-05-08T16:36:24.518 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-05-08T16:36:24.518 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-05-08T16:36:24.518 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-08T16:36:24.518 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-08T16:36:24.518 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-08T16:36:24.518 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T16:36:24.518 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81110 2024-05-08T16:36:24.518 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.81110/$cluster-$name.asok' 2024-05-08T16:36:24.518 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.81110/$cluster-$name.asok' 2024-05-08T16:36:24.518 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-05-08T16:36:24.518 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-05-08T16:36:24.518 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-05-08T16:36:24.518 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-05-08T16:36:24.518 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-05-08T16:36:24.518 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-08T16:36:24.519 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-08T16:36:24.519 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-08T16:36:24.519 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-08T16:36:24.519 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-05-08T16:36:24.519 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-05-08T16:36:24.519 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/2 2024-05-08T16:36:24.520 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-05-08T16:36:24.520 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=865dcd89-d07a-4aa3-939b-40ac9da53e8c 2024-05-08T16:36:24.520 INFO:tasks.workunit.client.0.smithi172.stdout:add osd2 865dcd89-d07a-4aa3-939b-40ac9da53e8c 2024-05-08T16:36:24.521 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 865dcd89-d07a-4aa3-939b-40ac9da53e8c' 2024-05-08T16:36:24.521 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-05-08T16:36:24.546 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAIqjtm/7TVIBAA+EvnHOxy0Um09o/Ge4tSLw== 2024-05-08T16:36:24.546 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAIqjtm/7TVIBAA+EvnHOxy0Um09o/Ge4tSLw=="}' 2024-05-08T16:36:24.546 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 865dcd89-d07a-4aa3-939b-40ac9da53e8c -i td/test-erasure-eio/2/new.json 2024-05-08T16:36:25.105 INFO:tasks.workunit.client.0.smithi172.stdout:2 2024-05-08T16:36:25.121 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/2/new.json 2024-05-08T16:36:25.123 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=4457b0f1-028b-482b-8ee3-7ee05861e6cc --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81110/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAIqjtm/7TVIBAA+EvnHOxy0Um09o/Ge4tSLw== --osd-uuid 865dcd89-d07a-4aa3-939b-40ac9da53e8c 2024-05-08T16:36:25.161 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:36:25.166+0000 7f98552a27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:36:25.163 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:36:25.166+0000 7f98552a27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:36:25.164 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:36:25.166+0000 7f98552a27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:36:25.165 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:36:25.166+0000 7f98552a27c0 -1 bluestore(td/test-erasure-eio/2/block) _read_bdev_label failed to open td/test-erasure-eio/2/block: (2) No such file or directory 2024-05-08T16:36:25.165 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:36:25.166+0000 7f98552a27c0 -1 bluestore(td/test-erasure-eio/2/block) _read_bdev_label failed to open td/test-erasure-eio/2/block: (2) No such file or directory 2024-05-08T16:36:25.165 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:36:25.166+0000 7f98552a27c0 -1 bluestore(td/test-erasure-eio/2/block) _read_bdev_label failed to open td/test-erasure-eio/2/block: (2) No such file or directory 2024-05-08T16:36:25.296 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:36:25.298+0000 7f98552a27c0 -1 bluestore(td/test-erasure-eio/2) _read_fsid unparsable uuid 2024-05-08T16:36:32.275 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/2/keyring 2024-05-08T16:36:32.275 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-05-08T16:36:32.276 INFO:tasks.workunit.client.0.smithi172.stdout:adding osd2 key to auth repository 2024-05-08T16:36:32.276 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-05-08T16:36:32.276 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-05-08T16:36:32.839 INFO:tasks.workunit.client.0.smithi172.stdout:start osd.2 2024-05-08T16:36:32.840 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-05-08T16:36:32.840 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=4457b0f1-028b-482b-8ee3-7ee05861e6cc --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81110/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-05-08T16:36:32.840 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-05-08T16:36:32.840 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-05-08T16:36:32.840 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-05-08T16:36:32.875 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:36:32.878+0000 7fab43ab47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:36:32.877 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:36:32.882+0000 7fab43ab47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:36:32.878 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:36:32.882+0000 7fab43ab47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:36:33.236 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-05-08T16:36:33.236 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-08T16:36:33.236 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-05-08T16:36:33.236 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-08T16:36:33.236 INFO:tasks.workunit.client.0.smithi172.stdout:0 2024-05-08T16:36:33.237 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-08T16:36:33.237 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:36:33.237 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-08T16:36:33.237 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:36:33.237 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T16:36:33.451 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:36:33.453+0000 7fab43ab47c0 -1 Falling back to public interface 2024-05-08T16:36:33.664 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:36:34.666 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:36:34.666 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:36:34.666 INFO:tasks.workunit.client.0.smithi172.stdout:1 2024-05-08T16:36:34.667 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-08T16:36:34.667 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:36:34.667 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T16:36:35.098 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:36:36.100 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:36:36.100 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:36:36.100 INFO:tasks.workunit.client.0.smithi172.stdout:2 2024-05-08T16:36:36.101 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-08T16:36:36.101 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:36:36.101 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T16:36:36.134 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:36:36.137+0000 7fab43ab47c0 -1 osd.2 0 log_to_monitors true 2024-05-08T16:36:36.818 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:36:37.819 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:36:37.820 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:36:37.820 INFO:tasks.workunit.client.0.smithi172.stdout:3 2024-05-08T16:36:37.821 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-05-08T16:36:37.821 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:36:37.821 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T16:36:38.412 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:36:39.414 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:36:39.414 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:36:39.414 INFO:tasks.workunit.client.0.smithi172.stdout:4 2024-05-08T16:36:39.415 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-05-08T16:36:39.415 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:36:39.415 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T16:36:39.876 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:36:40.878 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:36:40.878 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:36:40.878 INFO:tasks.workunit.client.0.smithi172.stdout:5 2024-05-08T16:36:40.879 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-05-08T16:36:40.879 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:36:40.879 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T16:36:41.450 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:36:42.452 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:36:42.452 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:36:42.453 INFO:tasks.workunit.client.0.smithi172.stdout:6 2024-05-08T16:36:42.453 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-05-08T16:36:42.453 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:36:42.453 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T16:36:42.922 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:36:43.924 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:36:43.924 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:36:43.924 INFO:tasks.workunit.client.0.smithi172.stdout:7 2024-05-08T16:36:43.925 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-05-08T16:36:43.925 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:36:43.925 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T16:36:44.666 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:36:45.668 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:36:45.668 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:36:45.669 INFO:tasks.workunit.client.0.smithi172.stdout:8 2024-05-08T16:36:45.669 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-05-08T16:36:45.669 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:36:45.669 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T16:36:46.246 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:36:47.248 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:36:47.248 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:36:47.249 INFO:tasks.workunit.client.0.smithi172.stdout:9 2024-05-08T16:36:47.249 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-05-08T16:36:47.249 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:36:47.249 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T16:36:47.747 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:36:48.749 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:36:48.749 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:36:48.750 INFO:tasks.workunit.client.0.smithi172.stdout:10 2024-05-08T16:36:48.750 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-05-08T16:36:48.750 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:36:48.750 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T16:36:49.346 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:36:50.348 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:36:50.348 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:36:50.349 INFO:tasks.workunit.client.0.smithi172.stdout:11 2024-05-08T16:36:50.350 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-05-08T16:36:50.350 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:36:50.350 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T16:36:50.891 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:36:51.893 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:36:51.893 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:36:51.893 INFO:tasks.workunit.client.0.smithi172.stdout:12 2024-05-08T16:36:51.894 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-05-08T16:36:51.894 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:36:51.894 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T16:36:52.777 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:36:53.778 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:36:53.778 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:36:53.779 INFO:tasks.workunit.client.0.smithi172.stdout:13 2024-05-08T16:36:53.779 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-05-08T16:36:53.779 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:36:53.779 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T16:36:54.683 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:36:55.684 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:36:55.685 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:36:55.685 INFO:tasks.workunit.client.0.smithi172.stdout:14 2024-05-08T16:36:55.685 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-05-08T16:36:55.685 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:36:55.686 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T16:36:56.985 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:36:57.987 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:36:57.987 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:36:57.987 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-05-08T16:36:57.987 INFO:tasks.workunit.client.0.smithi172.stdout:15 2024-05-08T16:36:57.988 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:36:57.988 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T16:36:58.905 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:36:59.907 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:36:59.907 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:36:59.907 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-05-08T16:36:59.907 INFO:tasks.workunit.client.0.smithi172.stdout:16 2024-05-08T16:36:59.907 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:36:59.908 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T16:37:00.935 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:37:01.937 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:37:01.937 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:37:01.937 INFO:tasks.workunit.client.0.smithi172.stdout:17 2024-05-08T16:37:01.938 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-05-08T16:37:01.938 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:37:01.938 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T16:37:02.912 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:37:03.914 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:37:03.914 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:37:03.914 INFO:tasks.workunit.client.0.smithi172.stdout:18 2024-05-08T16:37:03.915 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-05-08T16:37:03.915 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:37:03.915 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T16:37:04.352 INFO:tasks.workunit.client.0.smithi172.stdout:osd.2 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/4061653614,v1:127.0.0.1:6819/4061653614] [v2:127.0.0.1:6820/4061653614,v1:127.0.0.1:6821/4061653614] exists,up 865dcd89-d07a-4aa3-939b-40ac9da53e8c 2024-05-08T16:37:04.355 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-08T16:37:04.355 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-08T16:37:04.355 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-08T16:37:04.356 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-05-08T16:37:04.356 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 3 2024-05-08T16:37:04.356 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-05-08T16:37:04.356 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-05-08T16:37:04.356 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2024-05-08T16:37:04.356 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-05-08T16:37:04.356 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/3 2024-05-08T16:37:04.356 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=4457b0f1-028b-482b-8ee3-7ee05861e6cc --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 ' 2024-05-08T16:37:04.356 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-08T16:37:04.356 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-05-08T16:37:04.356 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-08T16:37:04.356 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/3' 2024-05-08T16:37:04.357 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/3/journal' 2024-05-08T16:37:04.357 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-05-08T16:37:04.357 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-05-08T16:37:04.357 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-05-08T16:37:04.357 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-05-08T16:37:04.357 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-08T16:37:04.357 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-08T16:37:04.357 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-08T16:37:04.357 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T16:37:04.358 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81110 2024-05-08T16:37:04.358 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.81110/$cluster-$name.asok' 2024-05-08T16:37:04.358 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.81110/$cluster-$name.asok' 2024-05-08T16:37:04.358 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-05-08T16:37:04.358 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-05-08T16:37:04.358 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-05-08T16:37:04.359 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-05-08T16:37:04.359 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-05-08T16:37:04.359 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-08T16:37:04.359 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-08T16:37:04.359 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-08T16:37:04.359 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-08T16:37:04.359 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-05-08T16:37:04.359 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-05-08T16:37:04.360 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/3 2024-05-08T16:37:04.360 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-05-08T16:37:04.360 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=414be264-6a94-40b7-a777-475d3268f24a 2024-05-08T16:37:04.360 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 414be264-6a94-40b7-a777-475d3268f24a' 2024-05-08T16:37:04.361 INFO:tasks.workunit.client.0.smithi172.stdout:add osd3 414be264-6a94-40b7-a777-475d3268f24a 2024-05-08T16:37:04.361 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-05-08T16:37:04.385 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAwqjtmgUo+FxAAK9ER7xkr54BDT1K8ssgyCQ== 2024-05-08T16:37:04.386 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAwqjtmgUo+FxAAK9ER7xkr54BDT1K8ssgyCQ=="}' 2024-05-08T16:37:04.386 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 414be264-6a94-40b7-a777-475d3268f24a -i td/test-erasure-eio/3/new.json 2024-05-08T16:37:05.742 INFO:tasks.workunit.client.0.smithi172.stdout:3 2024-05-08T16:37:05.766 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/3/new.json 2024-05-08T16:37:05.767 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=4457b0f1-028b-482b-8ee3-7ee05861e6cc --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81110/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAwqjtmgUo+FxAAK9ER7xkr54BDT1K8ssgyCQ== --osd-uuid 414be264-6a94-40b7-a777-475d3268f24a 2024-05-08T16:37:05.809 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:37:05.809+0000 7f35d40757c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:37:05.865 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:37:05.865+0000 7f35d40757c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:37:05.867 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:37:05.869+0000 7f35d40757c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:37:05.867 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:37:05.869+0000 7f35d40757c0 -1 bluestore(td/test-erasure-eio/3/block) _read_bdev_label failed to open td/test-erasure-eio/3/block: (2) No such file or directory 2024-05-08T16:37:05.867 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:37:05.869+0000 7f35d40757c0 -1 bluestore(td/test-erasure-eio/3/block) _read_bdev_label failed to open td/test-erasure-eio/3/block: (2) No such file or directory 2024-05-08T16:37:05.868 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:37:05.869+0000 7f35d40757c0 -1 bluestore(td/test-erasure-eio/3/block) _read_bdev_label failed to open td/test-erasure-eio/3/block: (2) No such file or directory 2024-05-08T16:37:06.040 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:37:06.041+0000 7f35d40757c0 -1 bluestore(td/test-erasure-eio/3) _read_fsid unparsable uuid 2024-05-08T16:37:13.179 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/3/keyring 2024-05-08T16:37:13.179 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-05-08T16:37:13.180 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2024-05-08T16:37:13.180 INFO:tasks.workunit.client.0.smithi172.stdout:adding osd3 key to auth repository 2024-05-08T16:37:13.180 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-05-08T16:37:13.738 INFO:tasks.workunit.client.0.smithi172.stdout:start osd.3 2024-05-08T16:37:13.739 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2024-05-08T16:37:13.739 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-05-08T16:37:13.739 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-05-08T16:37:13.739 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-05-08T16:37:13.739 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=4457b0f1-028b-482b-8ee3-7ee05861e6cc --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81110/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-05-08T16:37:13.774 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:37:13.776+0000 7f49837ed7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:37:13.776 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:37:13.776+0000 7f49837ed7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:37:13.777 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:37:13.780+0000 7f49837ed7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:37:14.216 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2024-05-08T16:37:14.216 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-08T16:37:14.216 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-05-08T16:37:14.216 INFO:tasks.workunit.client.0.smithi172.stdout:0 2024-05-08T16:37:14.217 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-08T16:37:14.217 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-08T16:37:14.217 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:37:14.217 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-08T16:37:14.217 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:37:14.217 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T16:37:14.328 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:37:14.328+0000 7f49837ed7c0 -1 Falling back to public interface 2024-05-08T16:37:14.653 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:37:15.655 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:37:15.655 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:37:15.656 INFO:tasks.workunit.client.0.smithi172.stdout:1 2024-05-08T16:37:15.656 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-08T16:37:15.656 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:37:15.657 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T16:37:16.076 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:37:17.078 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:37:17.078 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:37:17.078 INFO:tasks.workunit.client.0.smithi172.stdout:2 2024-05-08T16:37:17.079 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-08T16:37:17.079 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:37:17.079 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T16:37:17.097 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:37:17.096+0000 7f49837ed7c0 -1 osd.3 0 log_to_monitors true 2024-05-08T16:37:17.509 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:37:18.511 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:37:18.511 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:37:18.511 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-05-08T16:37:18.511 INFO:tasks.workunit.client.0.smithi172.stdout:3 2024-05-08T16:37:18.512 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:37:18.512 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T16:37:19.274 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:37:20.276 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:37:20.276 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:37:20.277 INFO:tasks.workunit.client.0.smithi172.stdout:4 2024-05-08T16:37:20.277 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-05-08T16:37:20.277 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:37:20.277 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T16:37:20.785 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:37:21.787 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:37:21.787 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:37:21.788 INFO:tasks.workunit.client.0.smithi172.stdout:5 2024-05-08T16:37:21.788 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-05-08T16:37:21.788 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:37:21.788 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T16:37:22.646 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:37:23.648 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:37:23.648 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:37:23.648 INFO:tasks.workunit.client.0.smithi172.stdout:6 2024-05-08T16:37:23.649 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-05-08T16:37:23.649 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:37:23.649 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T16:37:24.316 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:37:25.318 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:37:25.319 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:37:25.319 INFO:tasks.workunit.client.0.smithi172.stdout:7 2024-05-08T16:37:25.319 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-05-08T16:37:25.320 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:37:25.320 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T16:37:25.826 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:37:26.828 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:37:26.829 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:37:26.829 INFO:tasks.workunit.client.0.smithi172.stdout:8 2024-05-08T16:37:26.829 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-05-08T16:37:26.829 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:37:26.830 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T16:37:27.333 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:37:28.334 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:37:28.335 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:37:28.335 INFO:tasks.workunit.client.0.smithi172.stdout:9 2024-05-08T16:37:28.335 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-05-08T16:37:28.336 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:37:28.336 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T16:37:28.827 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:37:29.829 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:37:29.829 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:37:29.829 INFO:tasks.workunit.client.0.smithi172.stdout:10 2024-05-08T16:37:29.830 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-05-08T16:37:29.830 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:37:29.830 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T16:37:30.328 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:37:31.330 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:37:31.330 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:37:31.330 INFO:tasks.workunit.client.0.smithi172.stdout:11 2024-05-08T16:37:31.331 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-05-08T16:37:31.331 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:37:31.331 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T16:37:31.789 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:37:32.791 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:37:32.791 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:37:32.791 INFO:tasks.workunit.client.0.smithi172.stdout:12 2024-05-08T16:37:32.792 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-05-08T16:37:32.792 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:37:32.792 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T16:37:33.463 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:37:34.464 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:37:34.465 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:37:34.465 INFO:tasks.workunit.client.0.smithi172.stdout:13 2024-05-08T16:37:34.466 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-05-08T16:37:34.466 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:37:34.466 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T16:37:35.380 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:37:36.381 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:37:36.382 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:37:36.382 INFO:tasks.workunit.client.0.smithi172.stdout:14 2024-05-08T16:37:36.382 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-05-08T16:37:36.382 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:37:36.383 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T16:37:36.969 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:37:37.970 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:37:37.970 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:37:37.971 INFO:tasks.workunit.client.0.smithi172.stdout:15 2024-05-08T16:37:37.971 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-05-08T16:37:37.971 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:37:37.971 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T16:37:39.190 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:37:40.192 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:37:40.193 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:37:40.193 INFO:tasks.workunit.client.0.smithi172.stdout:16 2024-05-08T16:37:40.194 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-05-08T16:37:40.194 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:37:40.194 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T16:37:41.228 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:37:42.230 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:37:42.230 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:37:42.230 INFO:tasks.workunit.client.0.smithi172.stdout:17 2024-05-08T16:37:42.231 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-05-08T16:37:42.231 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:37:42.231 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T16:37:43.235 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:37:44.237 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:37:44.237 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:37:44.237 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-05-08T16:37:44.238 INFO:tasks.workunit.client.0.smithi172.stdout:18 2024-05-08T16:37:44.238 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:37:44.238 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T16:37:45.125 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:37:46.127 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:37:46.127 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:37:46.127 INFO:tasks.workunit.client.0.smithi172.stdout:19 2024-05-08T16:37:46.128 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-05-08T16:37:46.128 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:37:46.128 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T16:37:46.750 INFO:tasks.workunit.client.0.smithi172.stdout:osd.3 up in weight 1 up_from 30 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/521943770,v1:127.0.0.1:6827/521943770] [v2:127.0.0.1:6828/521943770,v1:127.0.0.1:6829/521943770] exists,up 414be264-6a94-40b7-a777-475d3268f24a 2024-05-08T16:37:46.755 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-08T16:37:46.755 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-08T16:37:46.755 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-08T16:37:46.755 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-05-08T16:37:46.755 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 4 2024-05-08T16:37:46.755 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-05-08T16:37:46.755 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-05-08T16:37:46.755 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=4 2024-05-08T16:37:46.755 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-05-08T16:37:46.755 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/4 2024-05-08T16:37:46.755 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=4457b0f1-028b-482b-8ee3-7ee05861e6cc --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 ' 2024-05-08T16:37:46.755 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-08T16:37:46.756 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-05-08T16:37:46.756 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-08T16:37:46.756 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/4' 2024-05-08T16:37:46.756 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/4/journal' 2024-05-08T16:37:46.756 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-05-08T16:37:46.756 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-05-08T16:37:46.756 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-05-08T16:37:46.756 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-05-08T16:37:46.756 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-08T16:37:46.756 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-08T16:37:46.756 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-08T16:37:46.756 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T16:37:46.756 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81110 2024-05-08T16:37:46.756 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.81110/$cluster-$name.asok' 2024-05-08T16:37:46.756 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.81110/$cluster-$name.asok' 2024-05-08T16:37:46.757 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-05-08T16:37:46.757 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-05-08T16:37:46.757 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-05-08T16:37:46.757 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-05-08T16:37:46.757 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-05-08T16:37:46.757 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-08T16:37:46.757 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-08T16:37:46.757 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-08T16:37:46.757 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-08T16:37:46.757 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-05-08T16:37:46.757 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-05-08T16:37:46.757 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/4 2024-05-08T16:37:46.758 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-05-08T16:37:46.758 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=5aa28e4a-60ab-41af-a910-647079c8214f 2024-05-08T16:37:46.758 INFO:tasks.workunit.client.0.smithi172.stdout:add osd4 5aa28e4a-60ab-41af-a910-647079c8214f 2024-05-08T16:37:46.759 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd4 5aa28e4a-60ab-41af-a910-647079c8214f' 2024-05-08T16:37:46.759 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-05-08T16:37:46.786 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBaqjtmxAAWLxAA3xO+ZQmT4CuQw3pj308wcQ== 2024-05-08T16:37:46.786 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBaqjtmxAAWLxAA3xO+ZQmT4CuQw3pj308wcQ=="}' 2024-05-08T16:37:46.786 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 5aa28e4a-60ab-41af-a910-647079c8214f -i td/test-erasure-eio/4/new.json 2024-05-08T16:37:47.391 INFO:tasks.workunit.client.0.smithi172.stdout:4 2024-05-08T16:37:47.408 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/4/new.json 2024-05-08T16:37:47.409 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 4 --fsid=4457b0f1-028b-482b-8ee3-7ee05861e6cc --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/4 --osd-journal=td/test-erasure-eio/4/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81110/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBaqjtmxAAWLxAA3xO+ZQmT4CuQw3pj308wcQ== --osd-uuid 5aa28e4a-60ab-41af-a910-647079c8214f 2024-05-08T16:37:47.440 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:37:47.439+0000 7eff46e347c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:37:47.442 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:37:47.443+0000 7eff46e347c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:37:47.443 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:37:47.443+0000 7eff46e347c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:37:47.444 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:37:47.443+0000 7eff46e347c0 -1 bluestore(td/test-erasure-eio/4/block) _read_bdev_label failed to open td/test-erasure-eio/4/block: (2) No such file or directory 2024-05-08T16:37:47.444 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:37:47.443+0000 7eff46e347c0 -1 bluestore(td/test-erasure-eio/4/block) _read_bdev_label failed to open td/test-erasure-eio/4/block: (2) No such file or directory 2024-05-08T16:37:47.444 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:37:47.443+0000 7eff46e347c0 -1 bluestore(td/test-erasure-eio/4/block) _read_bdev_label failed to open td/test-erasure-eio/4/block: (2) No such file or directory 2024-05-08T16:37:47.629 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:37:47.627+0000 7eff46e347c0 -1 bluestore(td/test-erasure-eio/4) _read_fsid unparsable uuid 2024-05-08T16:37:54.858 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/4/keyring 2024-05-08T16:37:54.858 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-05-08T16:37:54.860 INFO:tasks.workunit.client.0.smithi172.stdout:adding osd4 key to auth repository 2024-05-08T16:37:54.860 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd4 key to auth repository 2024-05-08T16:37:54.860 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-05-08T16:37:55.415 INFO:tasks.workunit.client.0.smithi172.stdout:start osd.4 2024-05-08T16:37:55.416 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.4 2024-05-08T16:37:55.416 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 4 --fsid=4457b0f1-028b-482b-8ee3-7ee05861e6cc --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/4 --osd-journal=td/test-erasure-eio/4/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81110/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-05-08T16:37:55.416 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-05-08T16:37:55.416 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-05-08T16:37:55.416 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-05-08T16:37:55.451 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:37:55.451+0000 7f9f84dd07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:37:55.453 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:37:55.451+0000 7f9f84dd07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:37:55.454 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:37:55.455+0000 7f9f84dd07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:37:55.813 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 4 2024-05-08T16:37:55.813 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-08T16:37:55.813 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=4 2024-05-08T16:37:55.814 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-08T16:37:55.814 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-08T16:37:55.814 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:37:55.814 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-08T16:37:55.814 INFO:tasks.workunit.client.0.smithi172.stdout:0 2024-05-08T16:37:55.814 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-08T16:37:55.815 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:37:56.036 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:37:56.035+0000 7f9f84dd07c0 -1 Falling back to public interface 2024-05-08T16:37:56.284 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:37:57.286 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:37:57.286 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:37:57.287 INFO:tasks.workunit.client.0.smithi172.stdout:1 2024-05-08T16:37:57.287 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-08T16:37:57.287 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:37:57.287 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-08T16:37:57.696 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:37:58.697 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:37:58.697 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:37:58.698 INFO:tasks.workunit.client.0.smithi172.stdout:2 2024-05-08T16:37:58.698 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-08T16:37:58.698 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:37:58.698 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-08T16:37:58.892 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:37:58.895+0000 7f9f84dd07c0 -1 osd.4 0 log_to_monitors true 2024-05-08T16:37:59.156 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:38:00.158 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:38:00.158 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:38:00.158 INFO:tasks.workunit.client.0.smithi172.stdout:3 2024-05-08T16:38:00.159 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-05-08T16:38:00.159 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:38:00.159 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-08T16:38:00.721 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:38:01.723 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:38:01.723 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:38:01.723 INFO:tasks.workunit.client.0.smithi172.stdout:4 2024-05-08T16:38:01.723 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-05-08T16:38:01.724 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:38:01.724 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-08T16:38:02.703 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:38:03.704 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:38:03.704 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:38:03.705 INFO:tasks.workunit.client.0.smithi172.stdout:5 2024-05-08T16:38:03.705 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-05-08T16:38:03.705 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:38:03.705 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-08T16:38:04.205 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:38:05.207 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:38:05.207 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:38:05.208 INFO:tasks.workunit.client.0.smithi172.stdout:6 2024-05-08T16:38:05.208 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-05-08T16:38:05.208 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:38:05.208 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-08T16:38:05.646 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:38:06.648 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:38:06.648 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:38:06.648 INFO:tasks.workunit.client.0.smithi172.stdout:7 2024-05-08T16:38:06.649 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-05-08T16:38:06.649 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:38:06.649 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-08T16:38:07.145 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:38:08.146 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:38:08.147 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:38:08.147 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-05-08T16:38:08.147 INFO:tasks.workunit.client.0.smithi172.stdout:8 2024-05-08T16:38:08.147 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:38:08.147 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-08T16:38:08.727 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:38:09.728 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:38:09.729 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:38:09.729 INFO:tasks.workunit.client.0.smithi172.stdout:9 2024-05-08T16:38:09.729 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-05-08T16:38:09.730 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:38:09.730 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-08T16:38:10.266 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:38:11.268 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:38:11.268 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:38:11.268 INFO:tasks.workunit.client.0.smithi172.stdout:10 2024-05-08T16:38:11.269 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-05-08T16:38:11.269 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:38:11.269 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-08T16:38:11.906 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:38:12.907 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:38:12.907 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:38:12.908 INFO:tasks.workunit.client.0.smithi172.stdout:11 2024-05-08T16:38:12.908 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-05-08T16:38:12.908 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:38:12.908 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-08T16:38:13.463 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:38:14.465 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:38:14.465 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:38:14.465 INFO:tasks.workunit.client.0.smithi172.stdout:12 2024-05-08T16:38:14.466 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-05-08T16:38:14.466 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:38:14.466 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-08T16:38:15.085 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:38:16.087 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:38:16.087 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:38:16.087 INFO:tasks.workunit.client.0.smithi172.stdout:13 2024-05-08T16:38:16.088 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-05-08T16:38:16.088 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:38:16.088 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-08T16:38:16.568 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:38:17.569 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:38:17.569 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:38:17.570 INFO:tasks.workunit.client.0.smithi172.stdout:14 2024-05-08T16:38:17.570 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-05-08T16:38:17.570 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:38:17.571 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-08T16:38:18.373 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:38:19.374 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:38:19.375 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:38:19.375 INFO:tasks.workunit.client.0.smithi172.stdout:15 2024-05-08T16:38:19.375 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-05-08T16:38:19.375 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:38:19.376 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-08T16:38:20.053 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:38:21.055 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:38:21.055 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:38:21.056 INFO:tasks.workunit.client.0.smithi172.stdout:16 2024-05-08T16:38:21.056 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-05-08T16:38:21.056 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:38:21.056 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-08T16:38:21.845 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:38:22.846 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:38:22.847 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:38:22.847 INFO:tasks.workunit.client.0.smithi172.stdout:17 2024-05-08T16:38:22.848 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-05-08T16:38:22.848 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:38:22.848 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-08T16:38:23.750 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:38:24.752 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:38:24.752 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:38:24.752 INFO:tasks.workunit.client.0.smithi172.stdout:18 2024-05-08T16:38:24.752 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-05-08T16:38:24.753 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:38:24.753 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-08T16:38:25.609 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:38:26.611 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:38:26.611 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:38:26.611 INFO:tasks.workunit.client.0.smithi172.stdout:19 2024-05-08T16:38:26.612 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-05-08T16:38:26.612 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:38:26.612 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-08T16:38:27.173 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:38:28.174 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:38:28.174 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:38:28.174 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-05-08T16:38:28.174 INFO:tasks.workunit.client.0.smithi172.stdout:20 2024-05-08T16:38:28.175 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:38:28.175 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-08T16:38:28.622 INFO:tasks.workunit.client.0.smithi172.stdout:osd.4 up in weight 1 up_from 41 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/2603129725,v1:127.0.0.1:6835/2603129725] [v2:127.0.0.1:6836/2603129725,v1:127.0.0.1:6837/2603129725] exists,up 5aa28e4a-60ab-41af-a910-647079c8214f 2024-05-08T16:38:28.627 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-08T16:38:28.627 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-08T16:38:28.627 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-08T16:38:28.627 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-05-08T16:38:28.627 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 5 2024-05-08T16:38:28.627 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-05-08T16:38:28.627 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-05-08T16:38:28.627 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=5 2024-05-08T16:38:28.627 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-05-08T16:38:28.627 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/5 2024-05-08T16:38:28.627 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=4457b0f1-028b-482b-8ee3-7ee05861e6cc --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 ' 2024-05-08T16:38:28.627 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-08T16:38:28.627 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-05-08T16:38:28.628 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-08T16:38:28.628 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/5' 2024-05-08T16:38:28.628 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/5/journal' 2024-05-08T16:38:28.628 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-05-08T16:38:28.628 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-05-08T16:38:28.628 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-05-08T16:38:28.628 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-05-08T16:38:28.628 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-08T16:38:28.628 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-08T16:38:28.628 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-08T16:38:28.628 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T16:38:28.628 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81110 2024-05-08T16:38:28.628 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.81110/$cluster-$name.asok' 2024-05-08T16:38:28.628 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.81110/$cluster-$name.asok' 2024-05-08T16:38:28.629 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-05-08T16:38:28.629 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-05-08T16:38:28.629 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-05-08T16:38:28.629 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-05-08T16:38:28.629 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-05-08T16:38:28.629 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-08T16:38:28.629 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-08T16:38:28.629 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-08T16:38:28.629 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-08T16:38:28.630 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-05-08T16:38:28.630 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-05-08T16:38:28.630 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/5 2024-05-08T16:38:28.630 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-05-08T16:38:28.630 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=cffe6f04-3a80-45d0-9c6c-9639bc43d4ce 2024-05-08T16:38:28.630 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd5 cffe6f04-3a80-45d0-9c6c-9639bc43d4ce' 2024-05-08T16:38:28.631 INFO:tasks.workunit.client.0.smithi172.stdout:add osd5 cffe6f04-3a80-45d0-9c6c-9639bc43d4ce 2024-05-08T16:38:28.631 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-05-08T16:38:28.656 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCEqjtmzcNTJxAAbEWY4ITu5t1lSgMg97XLuw== 2024-05-08T16:38:28.656 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCEqjtmzcNTJxAAbEWY4ITu5t1lSgMg97XLuw=="}' 2024-05-08T16:38:28.656 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new cffe6f04-3a80-45d0-9c6c-9639bc43d4ce -i td/test-erasure-eio/5/new.json 2024-05-08T16:38:30.325 INFO:tasks.workunit.client.0.smithi172.stdout:5 2024-05-08T16:38:30.348 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/5/new.json 2024-05-08T16:38:30.350 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 5 --fsid=4457b0f1-028b-482b-8ee3-7ee05861e6cc --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/5 --osd-journal=td/test-erasure-eio/5/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81110/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCEqjtmzcNTJxAAbEWY4ITu5t1lSgMg97XLuw== --osd-uuid cffe6f04-3a80-45d0-9c6c-9639bc43d4ce 2024-05-08T16:38:30.389 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:38:30.390+0000 7faf608127c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:38:30.391 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:38:30.394+0000 7faf608127c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:38:30.393 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:38:30.394+0000 7faf608127c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:38:30.393 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:38:30.394+0000 7faf608127c0 -1 bluestore(td/test-erasure-eio/5/block) _read_bdev_label failed to open td/test-erasure-eio/5/block: (2) No such file or directory 2024-05-08T16:38:30.393 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:38:30.394+0000 7faf608127c0 -1 bluestore(td/test-erasure-eio/5/block) _read_bdev_label failed to open td/test-erasure-eio/5/block: (2) No such file or directory 2024-05-08T16:38:30.393 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:38:30.394+0000 7faf608127c0 -1 bluestore(td/test-erasure-eio/5/block) _read_bdev_label failed to open td/test-erasure-eio/5/block: (2) No such file or directory 2024-05-08T16:38:30.542 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:38:30.546+0000 7faf608127c0 -1 bluestore(td/test-erasure-eio/5) _read_fsid unparsable uuid 2024-05-08T16:38:37.870 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/5/keyring 2024-05-08T16:38:37.872 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-05-08T16:38:37.872 INFO:tasks.workunit.client.0.smithi172.stdout:adding osd5 key to auth repository 2024-05-08T16:38:37.873 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd5 key to auth repository 2024-05-08T16:38:37.873 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/5/keyring auth add osd.5 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-05-08T16:38:38.441 INFO:tasks.workunit.client.0.smithi172.stdout:start osd.5 2024-05-08T16:38:38.442 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.5 2024-05-08T16:38:38.442 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 5 --fsid=4457b0f1-028b-482b-8ee3-7ee05861e6cc --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/5 --osd-journal=td/test-erasure-eio/5/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81110/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-05-08T16:38:38.442 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-05-08T16:38:38.442 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-05-08T16:38:38.442 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-05-08T16:38:38.477 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:38:38.478+0000 7fcfae7be7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:38:38.501 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:38:38.502+0000 7fcfae7be7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:38:38.503 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:38:38.506+0000 7fcfae7be7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:38:38.864 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 5 2024-05-08T16:38:38.864 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-08T16:38:38.864 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=5 2024-05-08T16:38:38.864 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-08T16:38:38.864 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-08T16:38:38.864 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:38:38.864 INFO:tasks.workunit.client.0.smithi172.stdout:0 2024-05-08T16:38:38.865 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-08T16:38:38.865 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:38:38.865 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-05-08T16:38:39.092 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:38:39.094+0000 7fcfae7be7c0 -1 Falling back to public interface 2024-05-08T16:38:39.299 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:38:40.301 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:38:40.301 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:38:40.302 INFO:tasks.workunit.client.0.smithi172.stdout:1 2024-05-08T16:38:40.302 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-08T16:38:40.302 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:38:40.302 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-05-08T16:38:40.746 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:38:41.748 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:38:41.748 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:38:41.748 INFO:tasks.workunit.client.0.smithi172.stdout:2 2024-05-08T16:38:41.749 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-08T16:38:41.749 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:38:41.749 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-05-08T16:38:41.782 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:38:41.786+0000 7fcfae7be7c0 -1 osd.5 0 log_to_monitors true 2024-05-08T16:38:42.173 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:38:43.175 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:38:43.175 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:38:43.175 INFO:tasks.workunit.client.0.smithi172.stdout:3 2024-05-08T16:38:43.176 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-05-08T16:38:43.176 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:38:43.176 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-05-08T16:38:43.715 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:38:44.717 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:38:44.717 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:38:44.717 INFO:tasks.workunit.client.0.smithi172.stdout:4 2024-05-08T16:38:44.718 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-05-08T16:38:44.718 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:38:44.718 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-05-08T16:38:45.447 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:38:46.449 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:38:46.450 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:38:46.450 INFO:tasks.workunit.client.0.smithi172.stdout:5 2024-05-08T16:38:46.450 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-05-08T16:38:46.450 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:38:46.450 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-05-08T16:38:47.027 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:38:48.029 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:38:48.029 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:38:48.029 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-05-08T16:38:48.029 INFO:tasks.workunit.client.0.smithi172.stdout:6 2024-05-08T16:38:48.030 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:38:48.030 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-05-08T16:38:48.551 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:38:49.553 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:38:49.553 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:38:49.554 INFO:tasks.workunit.client.0.smithi172.stdout:7 2024-05-08T16:38:49.554 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-05-08T16:38:49.554 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:38:49.554 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-05-08T16:38:50.194 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:38:51.197 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:38:51.197 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:38:51.197 INFO:tasks.workunit.client.0.smithi172.stdout:8 2024-05-08T16:38:51.198 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-05-08T16:38:51.198 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:38:51.198 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-05-08T16:38:51.619 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:38:52.621 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:38:52.621 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:38:52.621 INFO:tasks.workunit.client.0.smithi172.stdout:9 2024-05-08T16:38:52.622 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-05-08T16:38:52.622 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:38:52.622 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-05-08T16:38:53.123 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:38:54.124 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:38:54.124 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:38:54.124 INFO:tasks.workunit.client.0.smithi172.stdout:10 2024-05-08T16:38:54.125 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-05-08T16:38:54.125 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:38:54.125 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-05-08T16:38:54.668 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:38:55.669 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:38:55.669 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:38:55.670 INFO:tasks.workunit.client.0.smithi172.stdout:11 2024-05-08T16:38:55.670 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-05-08T16:38:55.670 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:38:55.670 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-05-08T16:38:56.226 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:38:57.227 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:38:57.228 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:38:57.228 INFO:tasks.workunit.client.0.smithi172.stdout:12 2024-05-08T16:38:57.228 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-05-08T16:38:57.229 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:38:57.229 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-05-08T16:38:58.075 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:38:59.077 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:38:59.078 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:38:59.078 INFO:tasks.workunit.client.0.smithi172.stdout:13 2024-05-08T16:38:59.079 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-05-08T16:38:59.079 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:38:59.080 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-05-08T16:38:59.875 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:39:00.876 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:39:00.877 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:39:00.877 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-05-08T16:39:00.877 INFO:tasks.workunit.client.0.smithi172.stdout:14 2024-05-08T16:39:00.877 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:39:00.877 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-05-08T16:39:01.759 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:39:02.760 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:39:02.760 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:39:02.761 INFO:tasks.workunit.client.0.smithi172.stdout:15 2024-05-08T16:39:02.761 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-05-08T16:39:02.761 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:39:02.761 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-05-08T16:39:03.639 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:39:04.641 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:39:04.641 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:39:04.641 INFO:tasks.workunit.client.0.smithi172.stdout:16 2024-05-08T16:39:04.642 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-05-08T16:39:04.642 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:39:04.642 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-05-08T16:39:05.702 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:39:06.703 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:39:06.704 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:39:06.704 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-05-08T16:39:06.704 INFO:tasks.workunit.client.0.smithi172.stdout:17 2024-05-08T16:39:06.704 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:39:06.704 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-05-08T16:39:07.631 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:39:08.628 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:39:08.628 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:39:08.629 INFO:tasks.workunit.client.0.smithi172.stdout:18 2024-05-08T16:39:08.629 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-05-08T16:39:08.629 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:39:08.629 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-05-08T16:39:09.463 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:39:10.465 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:39:10.465 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:39:10.465 INFO:tasks.workunit.client.0.smithi172.stdout:19 2024-05-08T16:39:10.466 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-05-08T16:39:10.466 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:39:10.466 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-05-08T16:39:10.946 INFO:tasks.workunit.client.0.smithi172.stdout:osd.5 up in weight 1 up_from 52 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6842/2054156471,v1:127.0.0.1:6843/2054156471] [v2:127.0.0.1:6844/2054156471,v1:127.0.0.1:6845/2054156471] exists,up cffe6f04-3a80-45d0-9c6c-9639bc43d4ce 2024-05-08T16:39:10.950 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-08T16:39:10.950 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-08T16:39:10.950 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-08T16:39:10.950 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-05-08T16:39:10.950 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 6 2024-05-08T16:39:10.950 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-05-08T16:39:10.950 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-05-08T16:39:10.950 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=6 2024-05-08T16:39:10.950 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-05-08T16:39:10.950 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/6 2024-05-08T16:39:10.951 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=4457b0f1-028b-482b-8ee3-7ee05861e6cc --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 ' 2024-05-08T16:39:10.951 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-08T16:39:10.951 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-05-08T16:39:10.951 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-08T16:39:10.951 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/6' 2024-05-08T16:39:10.951 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/6/journal' 2024-05-08T16:39:10.951 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-05-08T16:39:10.951 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-05-08T16:39:10.951 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-05-08T16:39:10.951 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-05-08T16:39:10.951 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-08T16:39:10.951 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-08T16:39:10.951 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-08T16:39:10.951 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T16:39:10.952 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81110 2024-05-08T16:39:10.952 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.81110/$cluster-$name.asok' 2024-05-08T16:39:10.952 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.81110/$cluster-$name.asok' 2024-05-08T16:39:10.952 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-05-08T16:39:10.952 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-05-08T16:39:10.952 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-05-08T16:39:10.952 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-05-08T16:39:10.952 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-05-08T16:39:10.952 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-08T16:39:10.952 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-08T16:39:10.952 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-08T16:39:10.952 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-08T16:39:10.952 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-05-08T16:39:10.952 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-05-08T16:39:10.952 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/6 2024-05-08T16:39:10.953 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-05-08T16:39:10.953 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=86390a39-779b-4f2e-9628-96f2fe4d90e1 2024-05-08T16:39:10.953 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd6 86390a39-779b-4f2e-9628-96f2fe4d90e1' 2024-05-08T16:39:10.954 INFO:tasks.workunit.client.0.smithi172.stdout:add osd6 86390a39-779b-4f2e-9628-96f2fe4d90e1 2024-05-08T16:39:10.954 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-05-08T16:39:10.973 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCuqjtmKjhCOhAA+z/0AleQRwUkY/YMJEmGAQ== 2024-05-08T16:39:10.973 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCuqjtmKjhCOhAA+z/0AleQRwUkY/YMJEmGAQ=="}' 2024-05-08T16:39:10.973 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 86390a39-779b-4f2e-9628-96f2fe4d90e1 -i td/test-erasure-eio/6/new.json 2024-05-08T16:39:11.832 INFO:tasks.workunit.client.0.smithi172.stdout:6 2024-05-08T16:39:11.852 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/6/new.json 2024-05-08T16:39:11.853 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 6 --fsid=4457b0f1-028b-482b-8ee3-7ee05861e6cc --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/6 --osd-journal=td/test-erasure-eio/6/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81110/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCuqjtmKjhCOhAA+z/0AleQRwUkY/YMJEmGAQ== --osd-uuid 86390a39-779b-4f2e-9628-96f2fe4d90e1 2024-05-08T16:39:11.885 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:39:11.885+0000 7fde6a8a77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:39:11.905 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:39:11.905+0000 7fde6a8a77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:39:11.906 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:39:11.909+0000 7fde6a8a77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:39:11.906 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:39:11.909+0000 7fde6a8a77c0 -1 bluestore(td/test-erasure-eio/6/block) _read_bdev_label failed to open td/test-erasure-eio/6/block: (2) No such file or directory 2024-05-08T16:39:11.906 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:39:11.909+0000 7fde6a8a77c0 -1 bluestore(td/test-erasure-eio/6/block) _read_bdev_label failed to open td/test-erasure-eio/6/block: (2) No such file or directory 2024-05-08T16:39:11.906 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:39:11.909+0000 7fde6a8a77c0 -1 bluestore(td/test-erasure-eio/6/block) _read_bdev_label failed to open td/test-erasure-eio/6/block: (2) No such file or directory 2024-05-08T16:39:12.006 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:39:12.009+0000 7fde6a8a77c0 -1 bluestore(td/test-erasure-eio/6) _read_fsid unparsable uuid 2024-05-08T16:39:19.738 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/6/keyring 2024-05-08T16:39:19.738 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-05-08T16:39:19.739 INFO:tasks.workunit.client.0.smithi172.stdout:adding osd6 key to auth repository 2024-05-08T16:39:19.739 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd6 key to auth repository 2024-05-08T16:39:19.739 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/6/keyring auth add osd.6 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-05-08T16:39:20.304 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.6 2024-05-08T16:39:20.304 INFO:tasks.workunit.client.0.smithi172.stdout:start osd.6 2024-05-08T16:39:20.305 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-05-08T16:39:20.305 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-05-08T16:39:20.305 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 6 --fsid=4457b0f1-028b-482b-8ee3-7ee05861e6cc --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/6 --osd-journal=td/test-erasure-eio/6/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81110/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-05-08T16:39:20.305 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-05-08T16:39:20.340 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:39:20.341+0000 7fa389a5d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:39:20.342 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:39:20.345+0000 7fa389a5d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:39:20.344 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:39:20.345+0000 7fa389a5d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:39:20.761 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 6 2024-05-08T16:39:20.761 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-08T16:39:20.761 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=6 2024-05-08T16:39:20.761 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-08T16:39:20.761 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-08T16:39:20.761 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:39:20.761 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-08T16:39:20.761 INFO:tasks.workunit.client.0.smithi172.stdout:0 2024-05-08T16:39:20.762 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:39:20.762 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-05-08T16:39:20.903 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:39:20.905+0000 7fa389a5d7c0 -1 Falling back to public interface 2024-05-08T16:39:21.209 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:39:22.210 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:39:22.211 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:39:22.211 INFO:tasks.workunit.client.0.smithi172.stdout:1 2024-05-08T16:39:22.211 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-08T16:39:22.212 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:39:22.212 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-05-08T16:39:22.664 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:39:23.636 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:39:23.637+0000 7fa389a5d7c0 -1 osd.6 0 log_to_monitors true 2024-05-08T16:39:23.665 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:39:23.665 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:39:23.665 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-08T16:39:23.666 INFO:tasks.workunit.client.0.smithi172.stdout:2 2024-05-08T16:39:23.666 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:39:23.666 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-05-08T16:39:24.456 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:39:25.458 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:39:25.458 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:39:25.458 INFO:tasks.workunit.client.0.smithi172.stdout:3 2024-05-08T16:39:25.459 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-05-08T16:39:25.459 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:39:25.459 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-05-08T16:39:25.942 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:39:26.944 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:39:26.944 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:39:26.944 INFO:tasks.workunit.client.0.smithi172.stdout:4 2024-05-08T16:39:26.945 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-05-08T16:39:26.945 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:39:26.945 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-05-08T16:39:27.557 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:39:28.559 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:39:28.559 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:39:28.559 INFO:tasks.workunit.client.0.smithi172.stdout:5 2024-05-08T16:39:28.559 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-05-08T16:39:28.560 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:39:28.560 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-05-08T16:39:29.179 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:39:30.181 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:39:30.181 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:39:30.181 INFO:tasks.workunit.client.0.smithi172.stdout:6 2024-05-08T16:39:30.181 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-05-08T16:39:30.182 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:39:30.182 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-05-08T16:39:30.695 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:39:31.696 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:39:31.696 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:39:31.697 INFO:tasks.workunit.client.0.smithi172.stdout:7 2024-05-08T16:39:31.697 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-05-08T16:39:31.697 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:39:31.697 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-05-08T16:39:32.203 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:39:33.204 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:39:33.204 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:39:33.205 INFO:tasks.workunit.client.0.smithi172.stdout:8 2024-05-08T16:39:33.205 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-05-08T16:39:33.205 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:39:33.206 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-05-08T16:39:33.755 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:39:34.757 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:39:34.757 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:39:34.757 INFO:tasks.workunit.client.0.smithi172.stdout:9 2024-05-08T16:39:34.758 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-05-08T16:39:34.758 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:39:34.758 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-05-08T16:39:35.203 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:39:36.204 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:39:36.204 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:39:36.205 INFO:tasks.workunit.client.0.smithi172.stdout:10 2024-05-08T16:39:36.205 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-05-08T16:39:36.205 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:39:36.205 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-05-08T16:39:36.725 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:39:37.727 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:39:37.727 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:39:37.727 INFO:tasks.workunit.client.0.smithi172.stdout:11 2024-05-08T16:39:37.728 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-05-08T16:39:37.728 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:39:37.728 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-05-08T16:39:38.203 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:39:39.205 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:39:39.205 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:39:39.205 INFO:tasks.workunit.client.0.smithi172.stdout:12 2024-05-08T16:39:39.206 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-05-08T16:39:39.206 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:39:39.207 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-05-08T16:39:40.142 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:39:41.144 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:39:41.144 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:39:41.145 INFO:tasks.workunit.client.0.smithi172.stdout:13 2024-05-08T16:39:41.145 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-05-08T16:39:41.145 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:39:41.145 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-05-08T16:39:42.072 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:39:43.074 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:39:43.074 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:39:43.074 INFO:tasks.workunit.client.0.smithi172.stdout:14 2024-05-08T16:39:43.075 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-05-08T16:39:43.075 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:39:43.075 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-05-08T16:39:43.964 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:39:44.966 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:39:44.966 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:39:44.966 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-05-08T16:39:44.966 INFO:tasks.workunit.client.0.smithi172.stdout:15 2024-05-08T16:39:44.967 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:39:44.967 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-05-08T16:39:46.021 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:39:47.023 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:39:47.023 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:39:47.024 INFO:tasks.workunit.client.0.smithi172.stdout:16 2024-05-08T16:39:47.025 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-05-08T16:39:47.025 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:39:47.025 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-05-08T16:39:48.234 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:39:49.236 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:39:49.236 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:39:49.236 INFO:tasks.workunit.client.0.smithi172.stdout:17 2024-05-08T16:39:49.237 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-05-08T16:39:49.237 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:39:49.237 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-05-08T16:39:50.237 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:39:51.238 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:39:51.239 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:39:51.239 INFO:tasks.workunit.client.0.smithi172.stdout:18 2024-05-08T16:39:51.239 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-05-08T16:39:51.239 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:39:51.240 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-05-08T16:39:51.819 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:39:52.821 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:39:52.821 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:39:52.821 INFO:tasks.workunit.client.0.smithi172.stdout:19 2024-05-08T16:39:52.822 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-05-08T16:39:52.822 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:39:52.822 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-05-08T16:39:53.371 INFO:tasks.workunit.client.0.smithi172.stdout:osd.6 up in weight 1 up_from 61 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6850/322994653,v1:127.0.0.1:6851/322994653] [v2:127.0.0.1:6852/322994653,v1:127.0.0.1:6853/322994653] exists,up 86390a39-779b-4f2e-9628-96f2fe4d90e1 2024-05-08T16:39:53.372 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-08T16:39:53.372 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-08T16:39:53.372 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-08T16:39:53.373 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: get_asok_path osd.0 2024-05-08T16:39:53.373 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-05-08T16:39:53.373 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-05-08T16:39:53.373 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-08T16:39:53.374 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T16:39:53.374 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81110 2024-05-08T16:39:53.374 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.81110/ceph-osd.0.asok 2024-05-08T16:39:53.374 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: CEPH_ARGS= 2024-05-08T16:39:53.374 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: ceph --admin-daemon /tmp/ceph-asok.81110/ceph-osd.0.asok log flush 2024-05-08T16:39:53.481 INFO:tasks.workunit.client.0.smithi172.stdout:{} 2024-05-08T16:39:53.495 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:56: setup_osds: grep 'load: jerasure.*lrc' td/test-erasure-eio/osd.0.log 2024-05-08T16:39:53.502 INFO:tasks.workunit.client.0.smithi172.stdout:2024-05-08T16:35:13.572+0000 7f31515817c0 0 load: jerasure load: lrc 2024-05-08T16:39:53.504 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:457: TEST_ec_recovery_multiple_objects: CEPH_ARGS='--fsid=4457b0f1-028b-482b-8ee3-7ee05861e6cc --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-05-08T16:39:53.504 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:459: TEST_ec_recovery_multiple_objects: local poolname=pool-jerasure 2024-05-08T16:39:53.504 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:460: TEST_ec_recovery_multiple_objects: create_erasure_coded_pool pool-jerasure 3 2 2024-05-08T16:39:53.504 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:67: create_erasure_coded_pool: local poolname=pool-jerasure 2024-05-08T16:39:53.504 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:68: create_erasure_coded_pool: shift 2024-05-08T16:39:53.504 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:69: create_erasure_coded_pool: local k=3 2024-05-08T16:39:53.504 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:70: create_erasure_coded_pool: shift 2024-05-08T16:39:53.504 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:71: create_erasure_coded_pool: local m=2 2024-05-08T16:39:53.505 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:72: create_erasure_coded_pool: shift 2024-05-08T16:39:53.505 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:74: create_erasure_coded_pool: ceph osd erasure-code-profile set myprofile plugin=jerasure k=3 m=2 crush-failure-domain=osd 2024-05-08T16:39:54.251 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:78: create_erasure_coded_pool: create_pool pool-jerasure 1 1 erasure myprofile 2024-05-08T16:39:54.251 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create pool-jerasure 1 1 erasure myprofile 2024-05-08T16:39:55.598 INFO:tasks.workunit.client.0.smithi172.stderr:pool 'pool-jerasure' already exists 2024-05-08T16:39:55.617 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-05-08T16:39:56.620 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:80: create_erasure_coded_pool: wait_for_clean 2024-05-08T16:39:56.620 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-05-08T16:39:56.620 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-05-08T16:39:56.620 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-05-08T16:39:56.620 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-05-08T16:39:56.621 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-05-08T16:39:56.621 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-05-08T16:39:56.622 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-05-08T16:39:56.622 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-05-08T16:39:56.622 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-05-08T16:39:56.743 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-05-08T16:39:56.743 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-05-08T16:39:56.743 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-05-08T16:39:56.743 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-05-08T16:39:56.743 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-05-08T16:39:56.743 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-05-08T16:39:57.197 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-05-08T16:39:57.197 INFO:tasks.workunit.client.0.smithi172.stderr:1 2024-05-08T16:39:57.198 INFO:tasks.workunit.client.0.smithi172.stderr:2 2024-05-08T16:39:57.198 INFO:tasks.workunit.client.0.smithi172.stderr:3 2024-05-08T16:39:57.198 INFO:tasks.workunit.client.0.smithi172.stderr:4 2024-05-08T16:39:57.198 INFO:tasks.workunit.client.0.smithi172.stderr:5 2024-05-08T16:39:57.198 INFO:tasks.workunit.client.0.smithi172.stderr:6' 2024-05-08T16:39:57.198 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-05-08T16:39:57.198 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T16:39:57.198 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-05-08T16:39:57.355 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=25769803831 2024-05-08T16:39:57.355 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 25769803831 2024-05-08T16:39:57.355 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803831' 2024-05-08T16:39:57.355 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T16:39:57.355 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-05-08T16:39:57.554 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=55834574898 2024-05-08T16:39:57.554 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 55834574898 2024-05-08T16:39:57.554 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803831 1-55834574898' 2024-05-08T16:39:57.554 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T16:39:57.554 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-05-08T16:39:57.699 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=85899345962 2024-05-08T16:39:57.699 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 85899345962 2024-05-08T16:39:57.699 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803831 1-55834574898 2-85899345962' 2024-05-08T16:39:57.699 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T16:39:57.699 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-05-08T16:39:57.849 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=128849018914 2024-05-08T16:39:57.849 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 128849018914 2024-05-08T16:39:57.850 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803831 1-55834574898 2-85899345962 3-128849018914' 2024-05-08T16:39:57.850 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T16:39:57.850 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-05-08T16:39:58.083 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=176093659162 2024-05-08T16:39:58.083 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 176093659162 2024-05-08T16:39:58.083 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803831 1-55834574898 2-85899345962 3-128849018914 4-176093659162' 2024-05-08T16:39:58.083 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T16:39:58.084 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-05-08T16:39:58.243 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=223338299407 2024-05-08T16:39:58.243 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 223338299407 2024-05-08T16:39:58.243 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803831 1-55834574898 2-85899345962 3-128849018914 4-176093659162 5-223338299407' 2024-05-08T16:39:58.243 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T16:39:58.244 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-05-08T16:39:58.401 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=261993005067 2024-05-08T16:39:58.401 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 261993005067 2024-05-08T16:39:58.401 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803831 1-55834574898 2-85899345962 3-128849018914 4-176093659162 5-223338299407 6-261993005067' 2024-05-08T16:39:58.401 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-08T16:39:58.402 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-08T16:39:58.402 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-25769803831 2024-05-08T16:39:58.402 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-05-08T16:39:58.403 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-25769803831 2024-05-08T16:39:58.403 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-08T16:39:58.404 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=25769803831 2024-05-08T16:39:58.404 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 25769803831' 2024-05-08T16:39:58.405 INFO:tasks.workunit.client.0.smithi172.stdout:waiting osd.0 seq 25769803831 2024-05-08T16:39:58.405 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-05-08T16:39:58.917 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 25769803831 -lt 25769803831 2024-05-08T16:39:58.917 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-08T16:39:58.918 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-55834574898 2024-05-08T16:39:58.918 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-08T16:39:58.919 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-05-08T16:39:58.920 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-55834574898 2024-05-08T16:39:58.920 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-08T16:39:58.921 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=55834574898 2024-05-08T16:39:58.921 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 55834574898' 2024-05-08T16:39:58.922 INFO:tasks.workunit.client.0.smithi172.stdout:waiting osd.1 seq 55834574898 2024-05-08T16:39:58.922 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-05-08T16:39:59.331 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 55834574898 -lt 55834574898 2024-05-08T16:39:59.331 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-08T16:39:59.332 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-08T16:39:59.332 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-85899345962 2024-05-08T16:39:59.333 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-05-08T16:39:59.334 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-85899345962 2024-05-08T16:39:59.334 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-08T16:39:59.335 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=85899345962 2024-05-08T16:39:59.335 INFO:tasks.workunit.client.0.smithi172.stdout:waiting osd.2 seq 85899345962 2024-05-08T16:39:59.335 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 85899345962' 2024-05-08T16:39:59.336 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-05-08T16:39:59.769 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 85899345962 -lt 85899345962 2024-05-08T16:39:59.769 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-08T16:39:59.770 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-128849018914 2024-05-08T16:39:59.770 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-08T16:39:59.771 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-05-08T16:39:59.772 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-128849018914 2024-05-08T16:39:59.772 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-08T16:39:59.774 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=128849018914 2024-05-08T16:39:59.774 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 128849018914' 2024-05-08T16:39:59.774 INFO:tasks.workunit.client.0.smithi172.stdout:waiting osd.3 seq 128849018914 2024-05-08T16:39:59.775 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-05-08T16:40:00.210 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 128849018914 -lt 128849018914 2024-05-08T16:40:00.210 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-08T16:40:00.211 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-176093659162 2024-05-08T16:40:00.211 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-08T16:40:00.211 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-05-08T16:40:00.212 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-176093659162 2024-05-08T16:40:00.213 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-08T16:40:00.214 INFO:tasks.workunit.client.0.smithi172.stdout:waiting osd.4 seq 176093659162 2024-05-08T16:40:00.215 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=176093659162 2024-05-08T16:40:00.215 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 176093659162' 2024-05-08T16:40:00.215 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-05-08T16:40:00.638 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 176093659162 -lt 176093659162 2024-05-08T16:40:00.638 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-08T16:40:00.639 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-223338299407 2024-05-08T16:40:00.639 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-08T16:40:00.641 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-05-08T16:40:00.642 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-223338299407 2024-05-08T16:40:00.642 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-08T16:40:00.643 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=223338299407 2024-05-08T16:40:00.643 INFO:tasks.workunit.client.0.smithi172.stdout:waiting osd.5 seq 223338299407 2024-05-08T16:40:00.644 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 223338299407' 2024-05-08T16:40:00.644 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-05-08T16:40:01.073 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 223338299407 -lt 223338299407 2024-05-08T16:40:01.073 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-08T16:40:01.073 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-261993005067 2024-05-08T16:40:01.073 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-08T16:40:01.074 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-05-08T16:40:01.075 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-261993005067 2024-05-08T16:40:01.075 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-08T16:40:01.077 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=261993005067 2024-05-08T16:40:01.077 INFO:tasks.workunit.client.0.smithi172.stdout:waiting osd.6 seq 261993005067 2024-05-08T16:40:01.077 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 261993005067' 2024-05-08T16:40:01.078 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-05-08T16:40:01.512 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 261993005067 -lt 261993005067 2024-05-08T16:40:01.512 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-05-08T16:40:01.513 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-05-08T16:40:01.513 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-05-08T16:40:02.138 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-05-08T16:40:02.138 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-05-08T16:40:02.138 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-05-08T16:40:02.139 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-05-08T16:40:02.139 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-05-08T16:40:02.139 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-05-08T16:40:02.139 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-05-08T16:40:02.139 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-05-08T16:40:02.592 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-05-08T16:40:02.592 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-05-08T16:40:02.593 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-05-08T16:40:02.593 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-05-08T16:40:03.130 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-05-08T16:40:03.130 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-05-08T16:40:03.130 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-05-08T16:40:03.130 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:462: TEST_ec_recovery_multiple_objects: rados_put td/test-erasure-eio pool-jerasure test1 2024-05-08T16:40:03.130 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:90: rados_put: local dir=td/test-erasure-eio 2024-05-08T16:40:03.131 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:91: rados_put: local poolname=pool-jerasure 2024-05-08T16:40:03.131 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:92: rados_put: local objname=test1 2024-05-08T16:40:03.131 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-05-08T16:40:03.131 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 AAA 2024-05-08T16:40:03.131 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-05-08T16:40:03.131 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 BBB 2024-05-08T16:40:03.131 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-05-08T16:40:03.131 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 CCCC 2024-05-08T16:40:03.131 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-05-08T16:40:03.131 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 DDDD 2024-05-08T16:40:03.131 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:100: rados_put: rados --pool pool-jerasure put test1 td/test-erasure-eio/ORIGINAL 2024-05-08T16:40:03.362 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:463: TEST_ec_recovery_multiple_objects: rados_put td/test-erasure-eio pool-jerasure test2 2024-05-08T16:40:03.362 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:90: rados_put: local dir=td/test-erasure-eio 2024-05-08T16:40:03.362 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:91: rados_put: local poolname=pool-jerasure 2024-05-08T16:40:03.362 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:92: rados_put: local objname=test2 2024-05-08T16:40:03.362 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-05-08T16:40:03.362 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 AAA 2024-05-08T16:40:03.363 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-05-08T16:40:03.363 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 BBB 2024-05-08T16:40:03.363 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-05-08T16:40:03.363 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 CCCC 2024-05-08T16:40:03.363 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-05-08T16:40:03.363 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 DDDD 2024-05-08T16:40:03.363 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:100: rados_put: rados --pool pool-jerasure put test2 td/test-erasure-eio/ORIGINAL 2024-05-08T16:40:03.680 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:464: TEST_ec_recovery_multiple_objects: rados_put td/test-erasure-eio pool-jerasure test3 2024-05-08T16:40:03.681 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:90: rados_put: local dir=td/test-erasure-eio 2024-05-08T16:40:03.681 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:91: rados_put: local poolname=pool-jerasure 2024-05-08T16:40:03.681 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:92: rados_put: local objname=test3 2024-05-08T16:40:03.681 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-05-08T16:40:03.681 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 AAA 2024-05-08T16:40:03.681 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-05-08T16:40:03.681 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 BBB 2024-05-08T16:40:03.681 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-05-08T16:40:03.681 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 CCCC 2024-05-08T16:40:03.681 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-05-08T16:40:03.681 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 DDDD 2024-05-08T16:40:03.681 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:100: rados_put: rados --pool pool-jerasure put test3 td/test-erasure-eio/ORIGINAL 2024-05-08T16:40:04.066 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:466: TEST_ec_recovery_multiple_objects: ceph osd out 0 2024-05-08T16:40:04.632 INFO:tasks.workunit.client.0.smithi172.stderr:osd.0 is already out. 2024-05-08T16:40:04.648 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:469: TEST_ec_recovery_multiple_objects: wait_for_clean 2024-05-08T16:40:04.649 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-05-08T16:40:04.649 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-05-08T16:40:04.649 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-05-08T16:40:04.649 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-05-08T16:40:04.649 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-05-08T16:40:04.649 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-05-08T16:40:04.649 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-05-08T16:40:04.650 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-05-08T16:40:04.650 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-05-08T16:40:04.753 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-05-08T16:40:04.753 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-05-08T16:40:04.753 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-05-08T16:40:04.754 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-05-08T16:40:04.754 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-05-08T16:40:04.754 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-05-08T16:40:05.594 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-05-08T16:40:05.594 INFO:tasks.workunit.client.0.smithi172.stderr:1 2024-05-08T16:40:05.594 INFO:tasks.workunit.client.0.smithi172.stderr:2 2024-05-08T16:40:05.594 INFO:tasks.workunit.client.0.smithi172.stderr:3 2024-05-08T16:40:05.594 INFO:tasks.workunit.client.0.smithi172.stderr:4 2024-05-08T16:40:05.594 INFO:tasks.workunit.client.0.smithi172.stderr:5 2024-05-08T16:40:05.594 INFO:tasks.workunit.client.0.smithi172.stderr:6' 2024-05-08T16:40:05.594 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-05-08T16:40:05.594 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T16:40:05.594 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-05-08T16:40:05.900 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=25769803835 2024-05-08T16:40:05.900 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 25769803835 2024-05-08T16:40:05.900 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803835' 2024-05-08T16:40:05.900 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T16:40:05.901 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-05-08T16:40:06.417 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=55834574902 2024-05-08T16:40:06.417 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 55834574902 2024-05-08T16:40:06.417 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803835 1-55834574902' 2024-05-08T16:40:06.417 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T16:40:06.417 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-05-08T16:40:06.617 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=85899345966 2024-05-08T16:40:06.617 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 85899345966 2024-05-08T16:40:06.617 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803835 1-55834574902 2-85899345966' 2024-05-08T16:40:06.618 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T16:40:06.618 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-05-08T16:40:06.882 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=128849018918 2024-05-08T16:40:06.882 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 128849018918 2024-05-08T16:40:06.882 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803835 1-55834574902 2-85899345966 3-128849018918' 2024-05-08T16:40:06.882 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T16:40:06.882 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-05-08T16:40:07.028 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=176093659165 2024-05-08T16:40:07.028 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 176093659165 2024-05-08T16:40:07.028 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803835 1-55834574902 2-85899345966 3-128849018918 4-176093659165' 2024-05-08T16:40:07.029 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T16:40:07.029 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-05-08T16:40:07.300 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=223338299411 2024-05-08T16:40:07.300 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 223338299411 2024-05-08T16:40:07.300 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803835 1-55834574902 2-85899345966 3-128849018918 4-176093659165 5-223338299411' 2024-05-08T16:40:07.300 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T16:40:07.300 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-05-08T16:40:07.454 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=261993005071 2024-05-08T16:40:07.454 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 261993005071 2024-05-08T16:40:07.454 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803835 1-55834574902 2-85899345966 3-128849018918 4-176093659165 5-223338299411 6-261993005071' 2024-05-08T16:40:07.454 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-08T16:40:07.454 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-25769803835 2024-05-08T16:40:07.455 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-08T16:40:07.456 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-05-08T16:40:07.457 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-25769803835 2024-05-08T16:40:07.457 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-08T16:40:07.458 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=25769803835 2024-05-08T16:40:07.458 INFO:tasks.workunit.client.0.smithi172.stdout:waiting osd.0 seq 25769803835 2024-05-08T16:40:07.461 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 25769803835' 2024-05-08T16:40:07.461 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-05-08T16:40:07.857 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 25769803835 -lt 25769803835 2024-05-08T16:40:07.857 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-08T16:40:07.858 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-55834574902 2024-05-08T16:40:07.858 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-08T16:40:07.859 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-05-08T16:40:07.859 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-55834574902 2024-05-08T16:40:07.859 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-08T16:40:07.861 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=55834574902 2024-05-08T16:40:07.861 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 55834574902' 2024-05-08T16:40:07.861 INFO:tasks.workunit.client.0.smithi172.stdout:waiting osd.1 seq 55834574902 2024-05-08T16:40:07.861 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-05-08T16:40:08.319 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 55834574902 -lt 55834574902 2024-05-08T16:40:08.319 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-08T16:40:08.320 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-08T16:40:08.320 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-85899345966 2024-05-08T16:40:08.321 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-05-08T16:40:08.322 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-85899345966 2024-05-08T16:40:08.322 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-08T16:40:08.323 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=85899345966 2024-05-08T16:40:08.323 INFO:tasks.workunit.client.0.smithi172.stdout:waiting osd.2 seq 85899345966 2024-05-08T16:40:08.324 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 85899345966' 2024-05-08T16:40:08.324 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-05-08T16:40:08.754 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 85899345966 -lt 85899345966 2024-05-08T16:40:08.754 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-08T16:40:08.754 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-128849018918 2024-05-08T16:40:08.755 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-08T16:40:08.755 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-05-08T16:40:08.757 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-128849018918 2024-05-08T16:40:08.757 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-08T16:40:08.757 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=128849018918 2024-05-08T16:40:08.757 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 128849018918' 2024-05-08T16:40:08.758 INFO:tasks.workunit.client.0.smithi172.stdout:waiting osd.3 seq 128849018918 2024-05-08T16:40:08.758 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-05-08T16:40:09.192 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 128849018918 -lt 128849018918 2024-05-08T16:40:09.193 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-08T16:40:09.193 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-176093659165 2024-05-08T16:40:09.193 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-08T16:40:09.195 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-05-08T16:40:09.195 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-176093659165 2024-05-08T16:40:09.196 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-08T16:40:09.196 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=176093659165 2024-05-08T16:40:09.197 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 176093659165' 2024-05-08T16:40:09.197 INFO:tasks.workunit.client.0.smithi172.stdout:waiting osd.4 seq 176093659165 2024-05-08T16:40:09.197 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-05-08T16:40:09.597 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 176093659166 -lt 176093659165 2024-05-08T16:40:09.598 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-08T16:40:09.598 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-223338299411 2024-05-08T16:40:09.598 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-08T16:40:09.599 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-05-08T16:40:09.600 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-223338299411 2024-05-08T16:40:09.600 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-08T16:40:09.601 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=223338299411 2024-05-08T16:40:09.601 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 223338299411' 2024-05-08T16:40:09.601 INFO:tasks.workunit.client.0.smithi172.stdout:waiting osd.5 seq 223338299411 2024-05-08T16:40:09.602 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-05-08T16:40:10.021 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 223338299411 -lt 223338299411 2024-05-08T16:40:10.021 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-08T16:40:10.022 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-261993005071 2024-05-08T16:40:10.022 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-08T16:40:10.023 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-05-08T16:40:10.024 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-261993005071 2024-05-08T16:40:10.024 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-08T16:40:10.025 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=261993005071 2024-05-08T16:40:10.026 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 261993005071' 2024-05-08T16:40:10.026 INFO:tasks.workunit.client.0.smithi172.stdout:waiting osd.6 seq 261993005071 2024-05-08T16:40:10.026 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-05-08T16:40:10.460 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 261993005071 -lt 261993005071 2024-05-08T16:40:10.460 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-05-08T16:40:10.461 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-05-08T16:40:10.461 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-05-08T16:40:11.065 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-05-08T16:40:11.066 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-05-08T16:40:11.067 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-05-08T16:40:11.067 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-05-08T16:40:11.067 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-05-08T16:40:11.067 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-05-08T16:40:11.067 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-05-08T16:40:11.067 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-05-08T16:40:11.489 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-05-08T16:40:11.490 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-05-08T16:40:11.490 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-05-08T16:40:11.490 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-05-08T16:40:12.043 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 5 2024-05-08T16:40:12.043 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' -1 2024-05-08T16:40:12.043 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-05-08T16:40:12.044 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=4 2024-05-08T16:40:12.044 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-05-08T16:40:12.044 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-05-08T16:40:12.144 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-05-08T16:40:12.144 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-05-08T16:40:12.145 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-05-08T16:40:12.145 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-05-08T16:40:12.146 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-05-08T16:40:12.146 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-05-08T16:40:12.146 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-05-08T16:40:12.146 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-05-08T16:40:12.537 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-05-08T16:40:12.537 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-05-08T16:40:12.538 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-05-08T16:40:12.538 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-05-08T16:40:13.072 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-05-08T16:40:13.072 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-05-08T16:40:13.072 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-05-08T16:40:13.072 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:471: TEST_ec_recovery_multiple_objects: rados_get td/test-erasure-eio pool-jerasure test1 2024-05-08T16:40:13.072 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2024-05-08T16:40:13.072 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2024-05-08T16:40:13.072 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=test1 2024-05-08T16:40:13.072 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=ok 2024-05-08T16:40:13.072 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' ok = fail ']' 2024-05-08T16:40:13.072 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:120: rados_get: rados --pool pool-jerasure get test1 td/test-erasure-eio/COPY 2024-05-08T16:40:13.205 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:121: rados_get: diff td/test-erasure-eio/ORIGINAL td/test-erasure-eio/COPY 2024-05-08T16:40:13.206 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:122: rados_get: rm td/test-erasure-eio/COPY 2024-05-08T16:40:13.207 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:472: TEST_ec_recovery_multiple_objects: rados_get td/test-erasure-eio pool-jerasure test2 2024-05-08T16:40:13.207 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2024-05-08T16:40:13.207 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2024-05-08T16:40:13.207 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=test2 2024-05-08T16:40:13.207 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=ok 2024-05-08T16:40:13.207 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' ok = fail ']' 2024-05-08T16:40:13.207 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:120: rados_get: rados --pool pool-jerasure get test2 td/test-erasure-eio/COPY 2024-05-08T16:40:13.256 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:121: rados_get: diff td/test-erasure-eio/ORIGINAL td/test-erasure-eio/COPY 2024-05-08T16:40:13.256 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:122: rados_get: rm td/test-erasure-eio/COPY 2024-05-08T16:40:13.258 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:473: TEST_ec_recovery_multiple_objects: rados_get td/test-erasure-eio pool-jerasure test3 2024-05-08T16:40:13.258 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2024-05-08T16:40:13.258 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2024-05-08T16:40:13.258 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=test3 2024-05-08T16:40:13.258 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=ok 2024-05-08T16:40:13.258 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' ok = fail ']' 2024-05-08T16:40:13.258 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:120: rados_get: rados --pool pool-jerasure get test3 td/test-erasure-eio/COPY 2024-05-08T16:40:13.287 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:121: rados_get: diff td/test-erasure-eio/ORIGINAL td/test-erasure-eio/COPY 2024-05-08T16:40:13.287 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:122: rados_get: rm td/test-erasure-eio/COPY 2024-05-08T16:40:13.288 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:475: TEST_ec_recovery_multiple_objects: delete_erasure_coded_pool pool-jerasure 2024-05-08T16:40:13.289 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:84: delete_erasure_coded_pool: local poolname=pool-jerasure 2024-05-08T16:40:13.289 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:85: delete_erasure_coded_pool: ceph osd pool delete pool-jerasure pool-jerasure --yes-i-really-really-mean-it 2024-05-08T16:40:13.781 INFO:tasks.workunit.client.0.smithi172.stderr:pool 'pool-jerasure' does not exist 2024-05-08T16:40:13.797 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:86: delete_erasure_coded_pool: ceph osd erasure-code-profile rm myprofile 2024-05-08T16:40:14.790 INFO:tasks.workunit.client.0.smithi172.stderr:erasure-code-profile myprofile does not exist 2024-05-08T16:40:14.811 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:42: run: teardown td/test-erasure-eio 2024-05-08T16:40:14.812 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-eio 2024-05-08T16:40:14.812 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-05-08T16:40:14.812 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-eio KILL 2024-05-08T16:40:14.813 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-08T16:40:14.813 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-08T16:40:14.813 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-08T16:40:14.813 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-08T16:40:14.813 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-08T16:40:14.926 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-08T16:40:14.926 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-05-08T16:40:14.927 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-05-08T16:40:14.927 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-05-08T16:40:14.928 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-05-08T16:40:14.928 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-05-08T16:40:14.928 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-05-08T16:40:14.929 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-08T16:40:14.929 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-05-08T16:40:14.930 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-05-08T16:40:14.930 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-08T16:40:14.930 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-05-08T16:40:14.931 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-05-08T16:40:14.932 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-eio 2024-05-08T16:40:14.968 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-05-08T16:40:14.968 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T16:40:14.968 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81110 2024-05-08T16:40:14.968 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.81110 2024-05-08T16:40:14.970 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-05-08T16:40:14.970 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-05-08T16:40:14.970 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:32: run: for func in $funcs 2024-05-08T16:40:14.970 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:33: run: setup td/test-erasure-eio 2024-05-08T16:40:14.971 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/test-erasure-eio 2024-05-08T16:40:14.971 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/test-erasure-eio 2024-05-08T16:40:14.971 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-eio 2024-05-08T16:40:14.971 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-05-08T16:40:14.971 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-eio KILL 2024-05-08T16:40:14.971 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-08T16:40:14.971 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-08T16:40:14.971 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-08T16:40:14.971 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-08T16:40:14.971 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-08T16:40:14.972 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-08T16:40:14.972 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-05-08T16:40:14.973 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-05-08T16:40:14.973 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-05-08T16:40:14.974 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-05-08T16:40:14.974 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-05-08T16:40:14.974 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-05-08T16:40:14.975 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-08T16:40:14.975 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-05-08T16:40:14.976 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-05-08T16:40:14.976 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-08T16:40:14.976 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-05-08T16:40:14.977 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-05-08T16:40:14.977 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-eio 2024-05-08T16:40:14.978 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-05-08T16:40:14.978 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T16:40:14.979 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81110 2024-05-08T16:40:14.979 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.81110 2024-05-08T16:40:14.979 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-05-08T16:40:14.980 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-05-08T16:40:14.980 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/test-erasure-eio 2024-05-08T16:40:14.980 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-05-08T16:40:14.981 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T16:40:14.981 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81110 2024-05-08T16:40:14.981 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.81110 2024-05-08T16:40:14.983 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-05-08T16:40:14.983 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-05-08T16:40:14.983 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-05-08T16:40:14.983 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/test-erasure-eio 1' TERM HUP INT 2024-05-08T16:40:14.983 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:34: run: run_mon td/test-erasure-eio a 2024-05-08T16:40:14.983 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/test-erasure-eio 2024-05-08T16:40:14.983 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-05-08T16:40:14.983 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-05-08T16:40:14.983 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-05-08T16:40:14.983 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/test-erasure-eio/a 2024-05-08T16:40:14.984 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-eio/a --run-dir=td/test-erasure-eio 2024-05-08T16:40:15.389 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-05-08T16:40:15.389 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-08T16:40:15.389 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-08T16:40:15.390 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-08T16:40:15.390 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T16:40:15.390 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81110 2024-05-08T16:40:15.390 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.81110/$cluster-$name.asok' 2024-05-08T16:40:15.390 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-eio/a '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.81110/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-eio/log --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-05-08T16:40:15.625 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-05-08T16:40:15.626 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-05-08T16:40:15.626 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-05-08T16:40:15.627 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-05-08T16:40:15.627 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-05-08T16:40:15.627 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-05-08T16:40:15.627 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-05-08T16:40:15.627 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-05-08T16:40:15.627 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-05-08T16:40:15.628 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-08T16:40:15.628 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T16:40:15.628 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81110 2024-05-08T16:40:15.629 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.81110/ceph-mon.a.asok 2024-05-08T16:40:15.629 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-05-08T16:40:15.629 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.81110/ceph-mon.a.asok config get fsid 2024-05-08T16:40:15.745 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-05-08T16:40:15.745 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-05-08T16:40:15.745 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-05-08T16:40:15.745 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-05-08T16:40:15.746 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-05-08T16:40:15.746 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-05-08T16:40:15.746 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-05-08T16:40:15.746 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-05-08T16:40:15.746 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-08T16:40:15.746 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T16:40:15.747 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81110 2024-05-08T16:40:15.747 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.81110/ceph-mon.a.asok 2024-05-08T16:40:15.747 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-05-08T16:40:15.747 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.81110/ceph-mon.a.asok config get mon_host 2024-05-08T16:40:15.856 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:35: run: run_mgr td/test-erasure-eio x 2024-05-08T16:40:15.857 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/test-erasure-eio 2024-05-08T16:40:15.857 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-05-08T16:40:15.857 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-05-08T16:40:15.857 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-05-08T16:40:15.857 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/test-erasure-eio/x 2024-05-08T16:40:15.857 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-05-08T16:40:16.788 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-05-08T16:40:16.788 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-08T16:40:16.788 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-08T16:40:16.789 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-08T16:40:16.789 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T16:40:16.789 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81110 2024-05-08T16:40:16.789 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.81110/$cluster-$name.asok' 2024-05-08T16:40:16.790 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-05-08T16:40:16.790 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-eio/x '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.81110/$cluster-$name.asok' --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-05-08T16:40:16.825 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:36: run: create_pool rbd 4 2024-05-08T16:40:16.825 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-05-08T16:40:17.203 INFO:tasks.workunit.client.0.smithi172.stderr:pool 'rbd' already exists 2024-05-08T16:40:17.221 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-05-08T16:40:18.223 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: get_asok_path mon.a 2024-05-08T16:40:18.224 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-05-08T16:40:18.224 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-05-08T16:40:18.225 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-08T16:40:18.225 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T16:40:18.225 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81110 2024-05-08T16:40:18.225 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.81110/ceph-mon.a.asok 2024-05-08T16:40:18.225 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: CEPH_ARGS= 2024-05-08T16:40:18.225 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: ceph --admin-daemon /tmp/ceph-asok.81110/ceph-mon.a.asok log flush 2024-05-08T16:40:18.324 INFO:tasks.workunit.client.0.smithi172.stdout:{} 2024-05-08T16:40:18.337 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:40: run: grep 'load: jerasure.*lrc' td/test-erasure-eio/mon.a.log 2024-05-08T16:40:18.339 INFO:tasks.workunit.client.0.smithi172.stdout:2024-05-08T16:40:15.432+0000 7f93a1513c80 0 load: jerasure load: lrc 2024-05-08T16:40:18.340 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:41: run: TEST_ec_recovery_multiple_objects_eio td/test-erasure-eio 2024-05-08T16:40:18.340 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:480: TEST_ec_recovery_multiple_objects_eio: local dir=td/test-erasure-eio 2024-05-08T16:40:18.340 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:481: TEST_ec_recovery_multiple_objects_eio: local objname=myobject 2024-05-08T16:40:18.340 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:483: TEST_ec_recovery_multiple_objects_eio: ORIG_ARGS='--fsid=4457b0f1-028b-482b-8ee3-7ee05861e6cc --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-05-08T16:40:18.340 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:484: TEST_ec_recovery_multiple_objects_eio: CEPH_ARGS+=' --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 ' 2024-05-08T16:40:18.340 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:485: TEST_ec_recovery_multiple_objects_eio: setup_osds 7 2024-05-08T16:40:18.340 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:47: setup_osds: local count=7 2024-05-08T16:40:18.340 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:48: setup_osds: shift 2024-05-08T16:40:18.341 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: expr 7 - 1 2024-05-08T16:40:18.342 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: seq 0 6 2024-05-08T16:40:18.344 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-05-08T16:40:18.344 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 0 2024-05-08T16:40:18.344 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-05-08T16:40:18.344 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-05-08T16:40:18.344 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-05-08T16:40:18.345 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-05-08T16:40:18.345 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/0 2024-05-08T16:40:18.345 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=4457b0f1-028b-482b-8ee3-7ee05861e6cc --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 ' 2024-05-08T16:40:18.345 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-08T16:40:18.345 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-05-08T16:40:18.345 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-08T16:40:18.345 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/0' 2024-05-08T16:40:18.345 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/0/journal' 2024-05-08T16:40:18.345 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-05-08T16:40:18.345 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-05-08T16:40:18.345 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-05-08T16:40:18.345 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-05-08T16:40:18.345 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-08T16:40:18.345 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-08T16:40:18.346 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-08T16:40:18.346 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T16:40:18.346 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81110 2024-05-08T16:40:18.346 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.81110/$cluster-$name.asok' 2024-05-08T16:40:18.347 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.81110/$cluster-$name.asok' 2024-05-08T16:40:18.347 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-05-08T16:40:18.347 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-05-08T16:40:18.347 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-05-08T16:40:18.347 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-05-08T16:40:18.348 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-05-08T16:40:18.348 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-08T16:40:18.348 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-08T16:40:18.348 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-08T16:40:18.348 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-08T16:40:18.348 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-05-08T16:40:18.348 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-05-08T16:40:18.348 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/0 2024-05-08T16:40:18.348 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-05-08T16:40:18.348 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=6a97927a-47d3-4ebe-97a3-c0dfa7523bb5 2024-05-08T16:40:18.348 INFO:tasks.workunit.client.0.smithi172.stdout:add osd0 6a97927a-47d3-4ebe-97a3-c0dfa7523bb5 2024-05-08T16:40:18.349 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 6a97927a-47d3-4ebe-97a3-c0dfa7523bb5' 2024-05-08T16:40:18.349 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-05-08T16:40:18.368 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDyqjtmv682FhAA25RdSQq+08ax4ytf0Mohdw== 2024-05-08T16:40:18.368 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDyqjtmv682FhAA25RdSQq+08ax4ytf0Mohdw=="}' 2024-05-08T16:40:18.368 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 6a97927a-47d3-4ebe-97a3-c0dfa7523bb5 -i td/test-erasure-eio/0/new.json 2024-05-08T16:40:18.704 INFO:tasks.workunit.client.0.smithi172.stdout:0 2024-05-08T16:40:18.719 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/0/new.json 2024-05-08T16:40:18.720 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=4457b0f1-028b-482b-8ee3-7ee05861e6cc --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81110/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDyqjtmv682FhAA25RdSQq+08ax4ytf0Mohdw== --osd-uuid 6a97927a-47d3-4ebe-97a3-c0dfa7523bb5 2024-05-08T16:40:18.752 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:40:18.752+0000 7fa5a7be97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:40:18.773 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:40:18.772+0000 7fa5a7be97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:40:18.775 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:40:18.776+0000 7fa5a7be97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:40:18.775 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:40:18.776+0000 7fa5a7be97c0 -1 bluestore(td/test-erasure-eio/0/block) _read_bdev_label failed to open td/test-erasure-eio/0/block: (2) No such file or directory 2024-05-08T16:40:18.775 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:40:18.776+0000 7fa5a7be97c0 -1 bluestore(td/test-erasure-eio/0/block) _read_bdev_label failed to open td/test-erasure-eio/0/block: (2) No such file or directory 2024-05-08T16:40:18.775 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:40:18.776+0000 7fa5a7be97c0 -1 bluestore(td/test-erasure-eio/0/block) _read_bdev_label failed to open td/test-erasure-eio/0/block: (2) No such file or directory 2024-05-08T16:40:18.813 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:40:18.812+0000 7fa5a7be97c0 -1 bluestore(td/test-erasure-eio/0) _read_fsid unparsable uuid 2024-05-08T16:40:26.030 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/0/keyring 2024-05-08T16:40:26.030 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-05-08T16:40:26.031 INFO:tasks.workunit.client.0.smithi172.stdout:adding osd0 key to auth repository 2024-05-08T16:40:26.031 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-05-08T16:40:26.031 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-05-08T16:40:26.606 INFO:tasks.workunit.client.0.smithi172.stdout:start osd.0 2024-05-08T16:40:26.606 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-05-08T16:40:26.606 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=4457b0f1-028b-482b-8ee3-7ee05861e6cc --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81110/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-05-08T16:40:26.606 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-05-08T16:40:26.607 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-05-08T16:40:26.607 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-05-08T16:40:26.641 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:40:26.639+0000 7f08497937c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:40:26.644 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:40:26.643+0000 7f08497937c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:40:26.645 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:40:26.647+0000 7f08497937c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:40:27.005 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-05-08T16:40:27.005 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-08T16:40:27.005 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-05-08T16:40:27.005 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-08T16:40:27.006 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-08T16:40:27.006 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:40:27.006 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-08T16:40:27.006 INFO:tasks.workunit.client.0.smithi172.stdout:0 2024-05-08T16:40:27.006 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:40:27.006 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:40:27.220 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:40:27.219+0000 7f08497937c0 -1 Falling back to public interface 2024-05-08T16:40:27.456 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:40:28.457 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:40:28.458 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:40:28.458 INFO:tasks.workunit.client.0.smithi172.stdout:1 2024-05-08T16:40:28.459 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-08T16:40:28.459 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:40:28.459 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:40:28.881 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:40:29.883 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:40:29.883 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:40:29.883 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-08T16:40:29.884 INFO:tasks.workunit.client.0.smithi172.stdout:2 2024-05-08T16:40:29.884 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:40:29.885 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:40:29.886 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:40:29.887+0000 7f08497937c0 -1 osd.0 0 log_to_monitors true 2024-05-08T16:40:30.509 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:40:31.510 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:40:31.510 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:40:31.510 INFO:tasks.workunit.client.0.smithi172.stdout:3 2024-05-08T16:40:31.511 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-05-08T16:40:31.511 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:40:31.511 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:40:32.097 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:40:33.099 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:40:33.099 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:40:33.099 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-05-08T16:40:33.100 INFO:tasks.workunit.client.0.smithi172.stdout:4 2024-05-08T16:40:33.100 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:40:33.100 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:40:33.590 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:40:34.591 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:40:34.592 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:40:34.592 INFO:tasks.workunit.client.0.smithi172.stdout:5 2024-05-08T16:40:34.592 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-05-08T16:40:34.592 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:40:34.593 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:40:35.075 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:40:36.077 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:40:36.077 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:40:36.078 INFO:tasks.workunit.client.0.smithi172.stdout:6 2024-05-08T16:40:36.078 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-05-08T16:40:36.078 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:40:36.078 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:40:36.525 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:40:37.527 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:40:37.527 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:40:37.527 INFO:tasks.workunit.client.0.smithi172.stdout:7 2024-05-08T16:40:37.528 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-05-08T16:40:37.528 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:40:37.528 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:40:38.004 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:40:39.005 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:40:39.005 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:40:39.005 INFO:tasks.workunit.client.0.smithi172.stdout:8 2024-05-08T16:40:39.006 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-05-08T16:40:39.006 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:40:39.006 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:40:39.495 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:40:40.497 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:40:40.497 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:40:40.497 INFO:tasks.workunit.client.0.smithi172.stdout:9 2024-05-08T16:40:40.498 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-05-08T16:40:40.498 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:40:40.498 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:40:41.071 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:40:42.073 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:40:42.073 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:40:42.073 INFO:tasks.workunit.client.0.smithi172.stdout:10 2024-05-08T16:40:42.074 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-05-08T16:40:42.074 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:40:42.074 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:40:42.600 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:40:43.601 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:40:43.601 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:40:43.602 INFO:tasks.workunit.client.0.smithi172.stdout:11 2024-05-08T16:40:43.602 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-05-08T16:40:43.602 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:40:43.602 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:40:44.114 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:40:45.115 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:40:45.116 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:40:45.116 INFO:tasks.workunit.client.0.smithi172.stdout:12 2024-05-08T16:40:45.116 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-05-08T16:40:45.117 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:40:45.117 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:40:45.754 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:40:46.756 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:40:46.756 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:40:46.757 INFO:tasks.workunit.client.0.smithi172.stdout:13 2024-05-08T16:40:46.757 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-05-08T16:40:46.757 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:40:46.757 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:40:47.875 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:40:48.877 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:40:48.877 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:40:48.877 INFO:tasks.workunit.client.0.smithi172.stdout:14 2024-05-08T16:40:48.878 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-05-08T16:40:48.878 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:40:48.878 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:40:49.868 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:40:50.870 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:40:50.870 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:40:50.870 INFO:tasks.workunit.client.0.smithi172.stdout:15 2024-05-08T16:40:50.871 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-05-08T16:40:50.871 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:40:50.871 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:40:52.099 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:40:53.100 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:40:53.100 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:40:53.101 INFO:tasks.workunit.client.0.smithi172.stdout:16 2024-05-08T16:40:53.101 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-05-08T16:40:53.101 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:40:53.101 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:40:54.278 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:40:55.280 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:40:55.280 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:40:55.280 INFO:tasks.workunit.client.0.smithi172.stdout:17 2024-05-08T16:40:55.281 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-05-08T16:40:55.281 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:40:55.281 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:40:56.304 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:40:57.306 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:40:57.306 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:40:57.306 INFO:tasks.workunit.client.0.smithi172.stdout:18 2024-05-08T16:40:57.307 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-05-08T16:40:57.307 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:40:57.307 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:40:57.770 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:40:58.772 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:40:58.772 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:40:58.772 INFO:tasks.workunit.client.0.smithi172.stdout:19 2024-05-08T16:40:58.773 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-05-08T16:40:58.773 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:40:58.773 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:40:59.292 INFO:tasks.workunit.client.0.smithi172.stdout:osd.0 up in weight 1 up_from 6 up_thru 6 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/655993362,v1:127.0.0.1:6803/655993362] [v2:127.0.0.1:6804/655993362,v1:127.0.0.1:6805/655993362] exists,up 6a97927a-47d3-4ebe-97a3-c0dfa7523bb5 2024-05-08T16:40:59.296 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-08T16:40:59.296 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-08T16:40:59.296 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-08T16:40:59.296 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-05-08T16:40:59.296 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 1 2024-05-08T16:40:59.296 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-05-08T16:40:59.296 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-05-08T16:40:59.297 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-05-08T16:40:59.297 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-05-08T16:40:59.297 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/1 2024-05-08T16:40:59.297 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=4457b0f1-028b-482b-8ee3-7ee05861e6cc --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 ' 2024-05-08T16:40:59.297 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-08T16:40:59.297 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-05-08T16:40:59.297 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-08T16:40:59.297 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/1' 2024-05-08T16:40:59.297 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/1/journal' 2024-05-08T16:40:59.297 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-05-08T16:40:59.297 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-05-08T16:40:59.297 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-05-08T16:40:59.297 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-05-08T16:40:59.297 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-08T16:40:59.297 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-08T16:40:59.298 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-08T16:40:59.298 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T16:40:59.298 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81110 2024-05-08T16:40:59.298 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.81110/$cluster-$name.asok' 2024-05-08T16:40:59.298 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.81110/$cluster-$name.asok' 2024-05-08T16:40:59.298 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-05-08T16:40:59.298 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-05-08T16:40:59.298 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-05-08T16:40:59.298 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-05-08T16:40:59.298 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-05-08T16:40:59.298 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-08T16:40:59.298 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-08T16:40:59.298 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-08T16:40:59.298 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-08T16:40:59.299 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-05-08T16:40:59.299 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-05-08T16:40:59.299 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/1 2024-05-08T16:40:59.299 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-05-08T16:40:59.299 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=2361c086-f13c-4ddd-b891-cd24615bebfc 2024-05-08T16:40:59.299 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 2361c086-f13c-4ddd-b891-cd24615bebfc' 2024-05-08T16:40:59.300 INFO:tasks.workunit.client.0.smithi172.stdout:add osd1 2361c086-f13c-4ddd-b891-cd24615bebfc 2024-05-08T16:40:59.300 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-05-08T16:40:59.323 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAbqztmrGaCExAAetLDOnbSD9zq17wq9qm4ZQ== 2024-05-08T16:40:59.323 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAbqztmrGaCExAAetLDOnbSD9zq17wq9qm4ZQ=="}' 2024-05-08T16:40:59.323 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 2361c086-f13c-4ddd-b891-cd24615bebfc -i td/test-erasure-eio/1/new.json 2024-05-08T16:40:59.916 INFO:tasks.workunit.client.0.smithi172.stdout:1 2024-05-08T16:40:59.933 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/1/new.json 2024-05-08T16:40:59.934 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=4457b0f1-028b-482b-8ee3-7ee05861e6cc --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81110/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAbqztmrGaCExAAetLDOnbSD9zq17wq9qm4ZQ== --osd-uuid 2361c086-f13c-4ddd-b891-cd24615bebfc 2024-05-08T16:40:59.972 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:40:59.975+0000 7f29e44837c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:40:59.974 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:40:59.979+0000 7f29e44837c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:40:59.976 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:40:59.979+0000 7f29e44837c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:40:59.976 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:40:59.979+0000 7f29e44837c0 -1 bluestore(td/test-erasure-eio/1/block) _read_bdev_label failed to open td/test-erasure-eio/1/block: (2) No such file or directory 2024-05-08T16:40:59.976 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:40:59.979+0000 7f29e44837c0 -1 bluestore(td/test-erasure-eio/1/block) _read_bdev_label failed to open td/test-erasure-eio/1/block: (2) No such file or directory 2024-05-08T16:40:59.976 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:40:59.979+0000 7f29e44837c0 -1 bluestore(td/test-erasure-eio/1/block) _read_bdev_label failed to open td/test-erasure-eio/1/block: (2) No such file or directory 2024-05-08T16:41:00.050 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:41:00.055+0000 7f29e44837c0 -1 bluestore(td/test-erasure-eio/1) _read_fsid unparsable uuid 2024-05-08T16:41:07.386 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/1/keyring 2024-05-08T16:41:07.387 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-05-08T16:41:07.388 INFO:tasks.workunit.client.0.smithi172.stdout:adding osd1 key to auth repository 2024-05-08T16:41:07.389 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-05-08T16:41:07.389 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-05-08T16:41:07.977 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-05-08T16:41:07.978 INFO:tasks.workunit.client.0.smithi172.stdout:start osd.1 2024-05-08T16:41:07.979 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=4457b0f1-028b-482b-8ee3-7ee05861e6cc --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81110/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-05-08T16:41:07.979 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-05-08T16:41:07.979 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-05-08T16:41:07.979 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-05-08T16:41:08.008 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:41:08.010+0000 7f895baed7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:41:08.010 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:41:08.014+0000 7f895baed7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:41:08.012 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:41:08.014+0000 7f895baed7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:41:08.373 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-05-08T16:41:08.373 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-08T16:41:08.373 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-05-08T16:41:08.373 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-08T16:41:08.374 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-08T16:41:08.374 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:41:08.374 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-08T16:41:08.374 INFO:tasks.workunit.client.0.smithi172.stdout:0 2024-05-08T16:41:08.374 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:41:08.374 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T16:41:08.555 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:41:08.558+0000 7f895baed7c0 -1 Falling back to public interface 2024-05-08T16:41:08.805 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:41:09.807 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:41:09.807 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:41:09.807 INFO:tasks.workunit.client.0.smithi172.stdout:1 2024-05-08T16:41:09.808 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-08T16:41:09.808 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:41:09.808 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T16:41:10.223 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:41:11.224 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:41:11.225 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:41:11.225 INFO:tasks.workunit.client.0.smithi172.stdout:2 2024-05-08T16:41:11.225 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-08T16:41:11.226 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:41:11.226 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T16:41:11.261 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:41:11.262+0000 7f895baed7c0 -1 osd.1 0 log_to_monitors true 2024-05-08T16:41:11.707 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:41:12.709 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:41:12.709 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:41:12.709 INFO:tasks.workunit.client.0.smithi172.stdout:3 2024-05-08T16:41:12.710 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-05-08T16:41:12.710 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:41:12.710 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T16:41:13.450 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:41:14.451 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:41:14.451 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:41:14.452 INFO:tasks.workunit.client.0.smithi172.stdout:4 2024-05-08T16:41:14.452 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-05-08T16:41:14.452 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:41:14.452 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T16:41:14.886 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:41:15.884 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:41:15.885 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:41:15.885 INFO:tasks.workunit.client.0.smithi172.stdout:5 2024-05-08T16:41:15.885 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-05-08T16:41:15.886 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:41:15.886 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T16:41:16.374 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:41:17.376 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:41:17.376 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:41:17.377 INFO:tasks.workunit.client.0.smithi172.stdout:6 2024-05-08T16:41:17.377 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-05-08T16:41:17.377 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:41:17.377 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T16:41:17.956 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:41:18.958 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:41:18.959 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:41:18.959 INFO:tasks.workunit.client.0.smithi172.stdout:7 2024-05-08T16:41:18.960 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-05-08T16:41:18.960 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:41:18.960 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T16:41:19.493 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:41:20.495 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:41:20.495 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:41:20.495 INFO:tasks.workunit.client.0.smithi172.stdout:8 2024-05-08T16:41:20.496 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-05-08T16:41:20.496 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:41:20.496 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T16:41:20.941 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:41:21.942 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:41:21.942 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:41:21.943 INFO:tasks.workunit.client.0.smithi172.stdout:9 2024-05-08T16:41:21.943 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-05-08T16:41:21.943 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:41:21.943 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T16:41:22.379 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:41:23.381 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:41:23.381 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:41:23.382 INFO:tasks.workunit.client.0.smithi172.stdout:10 2024-05-08T16:41:23.382 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-05-08T16:41:23.382 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:41:23.382 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T16:41:23.909 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:41:24.911 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:41:24.911 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:41:24.912 INFO:tasks.workunit.client.0.smithi172.stdout:11 2024-05-08T16:41:24.912 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-05-08T16:41:24.913 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:41:24.913 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T16:41:25.455 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:41:26.456 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:41:26.457 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:41:26.457 INFO:tasks.workunit.client.0.smithi172.stdout:12 2024-05-08T16:41:26.457 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-05-08T16:41:26.457 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:41:26.458 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T16:41:26.899 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:41:27.900 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:41:27.901 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:41:27.901 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-05-08T16:41:27.901 INFO:tasks.workunit.client.0.smithi172.stdout:13 2024-05-08T16:41:27.901 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:41:27.901 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T16:41:28.824 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:41:29.827 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:41:29.827 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:41:29.828 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-05-08T16:41:29.828 INFO:tasks.workunit.client.0.smithi172.stdout:14 2024-05-08T16:41:29.828 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:41:29.828 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T16:41:30.593 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:41:31.595 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:41:31.595 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:41:31.595 INFO:tasks.workunit.client.0.smithi172.stdout:15 2024-05-08T16:41:31.596 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-05-08T16:41:31.596 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:41:31.596 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T16:41:32.539 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:41:33.540 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:41:33.541 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:41:33.541 INFO:tasks.workunit.client.0.smithi172.stdout:16 2024-05-08T16:41:33.541 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-05-08T16:41:33.542 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:41:33.542 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T16:41:34.237 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:41:35.239 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:41:35.239 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:41:35.239 INFO:tasks.workunit.client.0.smithi172.stdout:17 2024-05-08T16:41:35.240 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-05-08T16:41:35.240 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:41:35.240 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T16:41:35.954 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:41:36.955 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:41:36.956 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:41:36.956 INFO:tasks.workunit.client.0.smithi172.stdout:18 2024-05-08T16:41:36.956 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-05-08T16:41:36.956 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:41:36.957 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T16:41:37.502 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:41:38.504 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:41:38.504 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:41:38.504 INFO:tasks.workunit.client.0.smithi172.stdout:19 2024-05-08T16:41:38.505 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-05-08T16:41:38.505 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:41:38.505 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T16:41:39.325 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:41:40.326 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:41:40.326 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:41:40.327 INFO:tasks.workunit.client.0.smithi172.stdout:20 2024-05-08T16:41:40.327 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-05-08T16:41:40.327 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:41:40.327 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T16:41:40.970 INFO:tasks.workunit.client.0.smithi172.stdout:osd.1 up in weight 1 up_from 13 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/1864115092,v1:127.0.0.1:6811/1864115092] [v2:127.0.0.1:6812/1864115092,v1:127.0.0.1:6813/1864115092] exists,up 2361c086-f13c-4ddd-b891-cd24615bebfc 2024-05-08T16:41:40.972 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-08T16:41:40.972 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-08T16:41:40.972 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-08T16:41:40.972 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-05-08T16:41:40.972 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 2 2024-05-08T16:41:40.972 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-05-08T16:41:40.972 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-05-08T16:41:40.972 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-05-08T16:41:40.972 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-05-08T16:41:40.972 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/2 2024-05-08T16:41:40.972 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=4457b0f1-028b-482b-8ee3-7ee05861e6cc --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 ' 2024-05-08T16:41:40.972 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-08T16:41:40.973 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-05-08T16:41:40.973 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-08T16:41:40.973 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/2' 2024-05-08T16:41:40.973 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/2/journal' 2024-05-08T16:41:40.973 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-05-08T16:41:40.973 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-05-08T16:41:40.973 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-05-08T16:41:40.973 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-05-08T16:41:40.973 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-08T16:41:40.973 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-08T16:41:40.975 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-08T16:41:40.975 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T16:41:40.975 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81110 2024-05-08T16:41:40.975 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.81110/$cluster-$name.asok' 2024-05-08T16:41:40.975 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.81110/$cluster-$name.asok' 2024-05-08T16:41:40.975 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-05-08T16:41:40.975 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-05-08T16:41:40.975 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-05-08T16:41:40.975 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-05-08T16:41:40.975 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-05-08T16:41:40.975 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-08T16:41:40.976 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-08T16:41:40.976 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-08T16:41:40.976 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-08T16:41:40.976 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-05-08T16:41:40.976 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-05-08T16:41:40.976 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/2 2024-05-08T16:41:40.976 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-05-08T16:41:40.976 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=ecde6cf9-2ebb-43b7-91d8-1bed40f72307 2024-05-08T16:41:40.976 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 ecde6cf9-2ebb-43b7-91d8-1bed40f72307' 2024-05-08T16:41:40.977 INFO:tasks.workunit.client.0.smithi172.stdout:add osd2 ecde6cf9-2ebb-43b7-91d8-1bed40f72307 2024-05-08T16:41:40.977 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-05-08T16:41:41.001 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBFqztmJfRWABAAj+DyiZAmtziU5HRmshNaNQ== 2024-05-08T16:41:41.001 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBFqztmJfRWABAAj+DyiZAmtziU5HRmshNaNQ=="}' 2024-05-08T16:41:41.001 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new ecde6cf9-2ebb-43b7-91d8-1bed40f72307 -i td/test-erasure-eio/2/new.json 2024-05-08T16:41:41.813 INFO:tasks.workunit.client.0.smithi172.stdout:2 2024-05-08T16:41:41.830 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/2/new.json 2024-05-08T16:41:41.831 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=4457b0f1-028b-482b-8ee3-7ee05861e6cc --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81110/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBFqztmJfRWABAAj+DyiZAmtziU5HRmshNaNQ== --osd-uuid ecde6cf9-2ebb-43b7-91d8-1bed40f72307 2024-05-08T16:41:41.871 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:41:41.874+0000 7ff9dfae17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:41:41.873 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:41:41.878+0000 7ff9dfae17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:41:41.874 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:41:41.878+0000 7ff9dfae17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:41:41.875 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:41:41.878+0000 7ff9dfae17c0 -1 bluestore(td/test-erasure-eio/2/block) _read_bdev_label failed to open td/test-erasure-eio/2/block: (2) No such file or directory 2024-05-08T16:41:41.875 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:41:41.878+0000 7ff9dfae17c0 -1 bluestore(td/test-erasure-eio/2/block) _read_bdev_label failed to open td/test-erasure-eio/2/block: (2) No such file or directory 2024-05-08T16:41:41.875 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:41:41.878+0000 7ff9dfae17c0 -1 bluestore(td/test-erasure-eio/2/block) _read_bdev_label failed to open td/test-erasure-eio/2/block: (2) No such file or directory 2024-05-08T16:41:42.048 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:41:42.050+0000 7ff9dfae17c0 -1 bluestore(td/test-erasure-eio/2) _read_fsid unparsable uuid 2024-05-08T16:41:49.598 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/2/keyring 2024-05-08T16:41:49.598 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-05-08T16:41:49.600 INFO:tasks.workunit.client.0.smithi172.stdout:adding osd2 key to auth repository 2024-05-08T16:41:49.600 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-05-08T16:41:49.600 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-05-08T16:41:50.190 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-05-08T16:41:50.190 INFO:tasks.workunit.client.0.smithi172.stdout:start osd.2 2024-05-08T16:41:50.191 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-05-08T16:41:50.191 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-05-08T16:41:50.191 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=4457b0f1-028b-482b-8ee3-7ee05861e6cc --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81110/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-05-08T16:41:50.191 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-05-08T16:41:50.227 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:41:50.230+0000 7f8c196347c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:41:50.263 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:41:50.266+0000 7f8c196347c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:41:50.265 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:41:50.266+0000 7f8c196347c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:41:50.647 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-05-08T16:41:50.647 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-08T16:41:50.647 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-05-08T16:41:50.647 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-08T16:41:50.647 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-08T16:41:50.647 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:41:50.648 INFO:tasks.workunit.client.0.smithi172.stdout:0 2024-05-08T16:41:50.648 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-08T16:41:50.648 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T16:41:50.648 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:41:50.843 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:41:50.846+0000 7f8c196347c0 -1 Falling back to public interface 2024-05-08T16:41:51.096 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:41:52.098 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:41:52.098 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:41:52.098 INFO:tasks.workunit.client.0.smithi172.stdout:1 2024-05-08T16:41:52.099 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-08T16:41:52.099 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:41:52.099 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T16:41:52.518 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:41:53.520 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:41:53.520 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:41:53.520 INFO:tasks.workunit.client.0.smithi172.stdout:2 2024-05-08T16:41:53.521 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-08T16:41:53.521 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:41:53.521 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T16:41:53.673 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:41:53.673+0000 7f8c196347c0 -1 osd.2 0 log_to_monitors true 2024-05-08T16:41:54.015 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:41:55.016 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:41:55.016 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:41:55.017 INFO:tasks.workunit.client.0.smithi172.stdout:3 2024-05-08T16:41:55.017 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-05-08T16:41:55.017 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:41:55.017 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T16:41:55.494 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:41:56.496 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:41:56.496 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:41:56.497 INFO:tasks.workunit.client.0.smithi172.stdout:4 2024-05-08T16:41:56.497 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-05-08T16:41:56.497 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:41:56.497 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T16:41:57.074 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:41:58.076 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:41:58.076 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:41:58.076 INFO:tasks.workunit.client.0.smithi172.stdout:5 2024-05-08T16:41:58.077 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-05-08T16:41:58.077 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:41:58.077 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T16:41:58.615 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:41:59.616 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:41:59.617 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:41:59.617 INFO:tasks.workunit.client.0.smithi172.stdout:6 2024-05-08T16:41:59.617 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-05-08T16:41:59.618 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:41:59.618 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T16:42:00.278 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:42:01.279 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:42:01.280 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:42:01.280 INFO:tasks.workunit.client.0.smithi172.stdout:7 2024-05-08T16:42:01.280 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-05-08T16:42:01.281 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:42:01.281 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T16:42:01.986 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:42:02.988 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:42:02.988 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:42:02.988 INFO:tasks.workunit.client.0.smithi172.stdout:8 2024-05-08T16:42:02.989 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-05-08T16:42:02.989 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:42:02.989 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T16:42:03.471 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:42:04.473 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:42:04.473 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:42:04.473 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-05-08T16:42:04.473 INFO:tasks.workunit.client.0.smithi172.stdout:9 2024-05-08T16:42:04.474 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:42:04.474 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T16:42:04.969 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:42:05.971 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:42:05.971 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:42:05.971 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-05-08T16:42:05.972 INFO:tasks.workunit.client.0.smithi172.stdout:10 2024-05-08T16:42:05.972 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:42:05.972 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T16:42:06.536 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:42:07.538 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:42:07.538 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:42:07.538 INFO:tasks.workunit.client.0.smithi172.stdout:11 2024-05-08T16:42:07.539 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-05-08T16:42:07.539 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:42:07.539 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T16:42:08.216 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:42:09.218 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:42:09.218 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:42:09.218 INFO:tasks.workunit.client.0.smithi172.stdout:12 2024-05-08T16:42:09.219 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-05-08T16:42:09.219 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:42:09.219 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T16:42:09.777 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:42:10.778 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:42:10.778 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:42:10.779 INFO:tasks.workunit.client.0.smithi172.stdout:13 2024-05-08T16:42:10.779 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-05-08T16:42:10.779 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:42:10.779 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T16:42:11.532 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:42:12.534 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:42:12.534 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:42:12.534 INFO:tasks.workunit.client.0.smithi172.stdout:14 2024-05-08T16:42:12.535 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-05-08T16:42:12.535 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:42:12.535 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T16:42:13.351 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:42:14.353 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:42:14.353 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:42:14.353 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-05-08T16:42:14.354 INFO:tasks.workunit.client.0.smithi172.stdout:15 2024-05-08T16:42:14.354 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:42:14.354 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T16:42:15.722 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:42:16.724 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:42:16.724 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:42:16.725 INFO:tasks.workunit.client.0.smithi172.stdout:16 2024-05-08T16:42:16.725 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-05-08T16:42:16.725 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:42:16.725 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T16:42:17.609 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:42:18.611 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:42:18.611 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:42:18.611 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-05-08T16:42:18.612 INFO:tasks.workunit.client.0.smithi172.stdout:17 2024-05-08T16:42:18.612 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:42:18.612 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T16:42:19.537 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:42:20.539 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:42:20.539 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:42:20.539 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-05-08T16:42:20.540 INFO:tasks.workunit.client.0.smithi172.stdout:18 2024-05-08T16:42:20.540 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:42:20.540 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T16:42:21.547 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:42:22.549 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:42:22.549 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:42:22.549 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-05-08T16:42:22.550 INFO:tasks.workunit.client.0.smithi172.stdout:19 2024-05-08T16:42:22.550 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:42:22.550 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T16:42:23.189 INFO:tasks.workunit.client.0.smithi172.stdout:osd.2 up in weight 1 up_from 19 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/3129036931,v1:127.0.0.1:6819/3129036931] [v2:127.0.0.1:6820/3129036931,v1:127.0.0.1:6821/3129036931] exists,up ecde6cf9-2ebb-43b7-91d8-1bed40f72307 2024-05-08T16:42:23.192 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-08T16:42:23.193 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-08T16:42:23.193 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-08T16:42:23.193 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-05-08T16:42:23.193 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 3 2024-05-08T16:42:23.193 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-05-08T16:42:23.193 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-05-08T16:42:23.193 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2024-05-08T16:42:23.193 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-05-08T16:42:23.193 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/3 2024-05-08T16:42:23.193 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=4457b0f1-028b-482b-8ee3-7ee05861e6cc --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 ' 2024-05-08T16:42:23.194 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-08T16:42:23.194 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-05-08T16:42:23.194 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-08T16:42:23.194 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/3' 2024-05-08T16:42:23.194 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/3/journal' 2024-05-08T16:42:23.194 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-05-08T16:42:23.194 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-05-08T16:42:23.194 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-05-08T16:42:23.194 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-05-08T16:42:23.194 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-08T16:42:23.194 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-08T16:42:23.195 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-08T16:42:23.195 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T16:42:23.195 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81110 2024-05-08T16:42:23.195 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.81110/$cluster-$name.asok' 2024-05-08T16:42:23.195 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.81110/$cluster-$name.asok' 2024-05-08T16:42:23.195 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-05-08T16:42:23.195 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-05-08T16:42:23.195 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-05-08T16:42:23.195 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-05-08T16:42:23.195 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-05-08T16:42:23.195 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-08T16:42:23.196 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-08T16:42:23.196 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-08T16:42:23.196 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-08T16:42:23.196 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-05-08T16:42:23.196 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-05-08T16:42:23.196 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/3 2024-05-08T16:42:23.197 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-05-08T16:42:23.197 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=679f3ef8-2634-440a-b91c-16f1bcd877ef 2024-05-08T16:42:23.197 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 679f3ef8-2634-440a-b91c-16f1bcd877ef' 2024-05-08T16:42:23.197 INFO:tasks.workunit.client.0.smithi172.stdout:add osd3 679f3ef8-2634-440a-b91c-16f1bcd877ef 2024-05-08T16:42:23.197 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-05-08T16:42:23.221 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBvqztmJSp/DRAAmSqsXdBlzK1KE64S6jtBog== 2024-05-08T16:42:23.222 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBvqztmJSp/DRAAmSqsXdBlzK1KE64S6jtBog=="}' 2024-05-08T16:42:23.222 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 679f3ef8-2634-440a-b91c-16f1bcd877ef -i td/test-erasure-eio/3/new.json 2024-05-08T16:42:24.182 INFO:tasks.workunit.client.0.smithi172.stdout:3 2024-05-08T16:42:24.200 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/3/new.json 2024-05-08T16:42:24.202 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=4457b0f1-028b-482b-8ee3-7ee05861e6cc --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81110/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBvqztmJSp/DRAAmSqsXdBlzK1KE64S6jtBog== --osd-uuid 679f3ef8-2634-440a-b91c-16f1bcd877ef 2024-05-08T16:42:24.248 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:42:24.249+0000 7f08f3fb27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:42:24.296 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:42:24.297+0000 7f08f3fb27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:42:24.298 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:42:24.301+0000 7f08f3fb27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:42:24.298 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:42:24.301+0000 7f08f3fb27c0 -1 bluestore(td/test-erasure-eio/3/block) _read_bdev_label failed to open td/test-erasure-eio/3/block: (2) No such file or directory 2024-05-08T16:42:24.298 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:42:24.301+0000 7f08f3fb27c0 -1 bluestore(td/test-erasure-eio/3/block) _read_bdev_label failed to open td/test-erasure-eio/3/block: (2) No such file or directory 2024-05-08T16:42:24.298 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:42:24.301+0000 7f08f3fb27c0 -1 bluestore(td/test-erasure-eio/3/block) _read_bdev_label failed to open td/test-erasure-eio/3/block: (2) No such file or directory 2024-05-08T16:42:24.451 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:42:24.453+0000 7f08f3fb27c0 -1 bluestore(td/test-erasure-eio/3) _read_fsid unparsable uuid 2024-05-08T16:42:32.008 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/3/keyring 2024-05-08T16:42:32.009 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-05-08T16:42:32.010 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2024-05-08T16:42:32.010 INFO:tasks.workunit.client.0.smithi172.stdout:adding osd3 key to auth repository 2024-05-08T16:42:32.011 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-05-08T16:42:32.578 INFO:tasks.workunit.client.0.smithi172.stdout:start osd.3 2024-05-08T16:42:32.579 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2024-05-08T16:42:32.579 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=4457b0f1-028b-482b-8ee3-7ee05861e6cc --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81110/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-05-08T16:42:32.579 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-05-08T16:42:32.579 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-05-08T16:42:32.579 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-05-08T16:42:32.614 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:42:32.617+0000 7f9df19c77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:42:32.615 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:42:32.617+0000 7f9df19c77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:42:32.617 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:42:32.617+0000 7f9df19c77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:42:32.979 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2024-05-08T16:42:32.979 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-08T16:42:32.979 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-05-08T16:42:32.980 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-08T16:42:32.980 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-08T16:42:32.980 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:42:32.980 INFO:tasks.workunit.client.0.smithi172.stdout:0 2024-05-08T16:42:32.980 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-08T16:42:32.981 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:42:32.981 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T16:42:33.159 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:42:33.161+0000 7f9df19c77c0 -1 Falling back to public interface 2024-05-08T16:42:33.430 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:42:34.432 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:42:34.432 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:42:34.432 INFO:tasks.workunit.client.0.smithi172.stdout:1 2024-05-08T16:42:34.433 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-08T16:42:34.433 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:42:34.433 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T16:42:34.850 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:42:35.851 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:42:35.851 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:42:35.851 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-08T16:42:35.852 INFO:tasks.workunit.client.0.smithi172.stdout:2 2024-05-08T16:42:35.852 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:42:35.852 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T16:42:35.977 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:42:35.957+0000 7f9df19c77c0 -1 osd.3 0 log_to_monitors true 2024-05-08T16:42:36.323 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:42:37.323 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:42:37.323 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:42:37.323 INFO:tasks.workunit.client.0.smithi172.stdout:3 2024-05-08T16:42:37.324 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-05-08T16:42:37.324 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:42:37.324 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T16:42:38.107 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:42:39.109 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:42:39.109 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:42:39.110 INFO:tasks.workunit.client.0.smithi172.stdout:4 2024-05-08T16:42:39.110 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-05-08T16:42:39.110 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:42:39.110 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T16:42:39.742 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:42:40.743 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:42:40.744 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:42:40.744 INFO:tasks.workunit.client.0.smithi172.stdout:5 2024-05-08T16:42:40.744 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-05-08T16:42:40.744 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:42:40.745 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T16:42:41.384 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:42:42.386 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:42:42.386 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:42:42.386 INFO:tasks.workunit.client.0.smithi172.stdout:6 2024-05-08T16:42:42.387 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-05-08T16:42:42.387 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:42:42.387 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T16:42:43.014 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:42:44.016 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:42:44.016 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:42:44.016 INFO:tasks.workunit.client.0.smithi172.stdout:7 2024-05-08T16:42:44.017 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-05-08T16:42:44.017 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:42:44.017 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T16:42:44.471 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:42:45.473 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:42:45.473 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:42:45.473 INFO:tasks.workunit.client.0.smithi172.stdout:8 2024-05-08T16:42:45.474 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-05-08T16:42:45.474 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:42:45.474 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T16:42:46.205 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:42:47.206 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:42:47.206 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:42:47.207 INFO:tasks.workunit.client.0.smithi172.stdout:9 2024-05-08T16:42:47.207 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-05-08T16:42:47.207 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:42:47.208 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T16:42:47.750 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:42:48.751 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:42:48.751 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:42:48.752 INFO:tasks.workunit.client.0.smithi172.stdout:10 2024-05-08T16:42:48.752 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-05-08T16:42:48.752 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T16:42:48.752 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:42:49.251 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:42:50.253 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:42:50.253 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:42:50.253 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-05-08T16:42:50.254 INFO:tasks.workunit.client.0.smithi172.stdout:11 2024-05-08T16:42:50.254 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:42:50.254 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T16:42:50.864 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:42:51.866 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:42:51.866 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:42:51.866 INFO:tasks.workunit.client.0.smithi172.stdout:12 2024-05-08T16:42:51.867 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-05-08T16:42:51.867 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:42:51.867 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T16:42:52.440 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:42:53.442 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:42:53.442 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:42:53.442 INFO:tasks.workunit.client.0.smithi172.stdout:13 2024-05-08T16:42:53.443 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-05-08T16:42:53.443 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:42:53.443 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T16:42:54.143 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:42:55.145 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:42:55.145 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:42:55.145 INFO:tasks.workunit.client.0.smithi172.stdout:14 2024-05-08T16:42:55.146 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-05-08T16:42:55.146 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:42:55.146 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T16:42:55.899 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:42:56.901 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:42:56.901 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:42:56.901 INFO:tasks.workunit.client.0.smithi172.stdout:15 2024-05-08T16:42:56.902 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-05-08T16:42:56.902 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:42:56.902 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T16:42:57.969 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:42:58.970 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:42:58.970 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:42:58.970 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-05-08T16:42:58.970 INFO:tasks.workunit.client.0.smithi172.stdout:16 2024-05-08T16:42:58.971 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:42:58.971 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T16:43:00.058 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:43:01.060 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:43:01.060 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:43:01.060 INFO:tasks.workunit.client.0.smithi172.stdout:17 2024-05-08T16:43:01.060 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-05-08T16:43:01.061 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:43:01.061 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T16:43:01.915 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:43:02.917 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:43:02.917 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:43:02.918 INFO:tasks.workunit.client.0.smithi172.stdout:18 2024-05-08T16:43:02.918 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-05-08T16:43:02.918 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:43:02.918 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T16:43:03.819 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:43:04.821 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:43:04.821 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:43:04.822 INFO:tasks.workunit.client.0.smithi172.stdout:19 2024-05-08T16:43:04.822 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-05-08T16:43:04.823 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:43:04.823 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T16:43:05.440 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:43:06.440 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:43:06.440 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:43:06.441 INFO:tasks.workunit.client.0.smithi172.stdout:20 2024-05-08T16:43:06.441 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-05-08T16:43:06.441 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:43:06.441 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T16:43:07.021 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:43:08.022 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:43:08.023 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:43:08.023 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-05-08T16:43:08.023 INFO:tasks.workunit.client.0.smithi172.stdout:21 2024-05-08T16:43:08.024 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:43:08.024 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T16:43:08.686 INFO:tasks.workunit.client.0.smithi172.stdout:osd.3 up in weight 1 up_from 28 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/2975402645,v1:127.0.0.1:6827/2975402645] [v2:127.0.0.1:6828/2975402645,v1:127.0.0.1:6829/2975402645] exists,up 679f3ef8-2634-440a-b91c-16f1bcd877ef 2024-05-08T16:43:08.689 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-08T16:43:08.689 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-08T16:43:08.689 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-08T16:43:08.689 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-05-08T16:43:08.689 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 4 2024-05-08T16:43:08.689 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-05-08T16:43:08.690 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-05-08T16:43:08.690 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=4 2024-05-08T16:43:08.690 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-05-08T16:43:08.690 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/4 2024-05-08T16:43:08.690 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=4457b0f1-028b-482b-8ee3-7ee05861e6cc --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 ' 2024-05-08T16:43:08.690 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-08T16:43:08.690 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-05-08T16:43:08.690 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-08T16:43:08.690 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/4' 2024-05-08T16:43:08.690 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/4/journal' 2024-05-08T16:43:08.690 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-05-08T16:43:08.690 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-05-08T16:43:08.690 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-05-08T16:43:08.691 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-05-08T16:43:08.691 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-08T16:43:08.691 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-08T16:43:08.691 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-08T16:43:08.691 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T16:43:08.691 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81110 2024-05-08T16:43:08.691 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.81110/$cluster-$name.asok' 2024-05-08T16:43:08.691 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.81110/$cluster-$name.asok' 2024-05-08T16:43:08.691 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-05-08T16:43:08.691 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-05-08T16:43:08.691 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-05-08T16:43:08.691 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-05-08T16:43:08.691 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-05-08T16:43:08.692 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-08T16:43:08.692 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-08T16:43:08.692 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-08T16:43:08.692 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-08T16:43:08.692 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-05-08T16:43:08.692 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-05-08T16:43:08.692 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/4 2024-05-08T16:43:08.692 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-05-08T16:43:08.693 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=7ed0bd11-b5df-4978-bf71-408483e65430 2024-05-08T16:43:08.693 INFO:tasks.workunit.client.0.smithi172.stdout:add osd4 7ed0bd11-b5df-4978-bf71-408483e65430 2024-05-08T16:43:08.693 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd4 7ed0bd11-b5df-4978-bf71-408483e65430' 2024-05-08T16:43:08.693 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-05-08T16:43:08.717 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCcqztmba0SKxAAJ9gXBoDqzvUtQnrTUP7xkA== 2024-05-08T16:43:08.717 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCcqztmba0SKxAAJ9gXBoDqzvUtQnrTUP7xkA=="}' 2024-05-08T16:43:08.718 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 7ed0bd11-b5df-4978-bf71-408483e65430 -i td/test-erasure-eio/4/new.json 2024-05-08T16:43:09.626 INFO:tasks.workunit.client.0.smithi172.stdout:4 2024-05-08T16:43:09.644 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/4/new.json 2024-05-08T16:43:09.644 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 4 --fsid=4457b0f1-028b-482b-8ee3-7ee05861e6cc --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/4 --osd-journal=td/test-erasure-eio/4/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81110/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCcqztmba0SKxAAJ9gXBoDqzvUtQnrTUP7xkA== --osd-uuid 7ed0bd11-b5df-4978-bf71-408483e65430 2024-05-08T16:43:09.683 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:43:09.684+0000 7f95afbff7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:43:09.736 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:43:09.736+0000 7f95afbff7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:43:09.737 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:43:09.736+0000 7f95afbff7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:43:09.737 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:43:09.740+0000 7f95afbff7c0 -1 bluestore(td/test-erasure-eio/4/block) _read_bdev_label failed to open td/test-erasure-eio/4/block: (2) No such file or directory 2024-05-08T16:43:09.737 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:43:09.740+0000 7f95afbff7c0 -1 bluestore(td/test-erasure-eio/4/block) _read_bdev_label failed to open td/test-erasure-eio/4/block: (2) No such file or directory 2024-05-08T16:43:09.737 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:43:09.740+0000 7f95afbff7c0 -1 bluestore(td/test-erasure-eio/4/block) _read_bdev_label failed to open td/test-erasure-eio/4/block: (2) No such file or directory 2024-05-08T16:43:09.927 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:43:09.928+0000 7f95afbff7c0 -1 bluestore(td/test-erasure-eio/4) _read_fsid unparsable uuid 2024-05-08T16:43:17.774 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/4/keyring 2024-05-08T16:43:17.774 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-05-08T16:43:17.775 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd4 key to auth repository 2024-05-08T16:43:17.775 INFO:tasks.workunit.client.0.smithi172.stdout:adding osd4 key to auth repository 2024-05-08T16:43:17.775 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-05-08T16:43:18.372 INFO:tasks.workunit.client.0.smithi172.stdout:start osd.4 2024-05-08T16:43:18.372 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.4 2024-05-08T16:43:18.373 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 4 --fsid=4457b0f1-028b-482b-8ee3-7ee05861e6cc --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/4 --osd-journal=td/test-erasure-eio/4/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81110/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-05-08T16:43:18.373 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-05-08T16:43:18.373 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-05-08T16:43:18.373 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-05-08T16:43:18.407 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:43:18.408+0000 7fd04a1f17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:43:18.409 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:43:18.412+0000 7fd04a1f17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:43:18.410 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:43:18.412+0000 7fd04a1f17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:43:18.847 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 4 2024-05-08T16:43:18.847 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-08T16:43:18.847 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=4 2024-05-08T16:43:18.847 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-08T16:43:18.847 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-08T16:43:18.847 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:43:18.847 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-08T16:43:18.848 INFO:tasks.workunit.client.0.smithi172.stdout:0 2024-05-08T16:43:18.848 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:43:18.848 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-08T16:43:18.990 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:43:18.992+0000 7fd04a1f17c0 -1 Falling back to public interface 2024-05-08T16:43:19.287 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:43:20.289 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:43:20.289 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:43:20.289 INFO:tasks.workunit.client.0.smithi172.stdout:1 2024-05-08T16:43:20.290 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-08T16:43:20.290 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:43:20.290 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-08T16:43:20.691 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:43:21.693 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:43:21.693 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:43:21.694 INFO:tasks.workunit.client.0.smithi172.stdout:2 2024-05-08T16:43:21.694 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-08T16:43:21.694 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:43:21.694 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-08T16:43:21.749 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:43:21.748+0000 7fd04a1f17c0 -1 osd.4 0 log_to_monitors true 2024-05-08T16:43:22.196 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:43:23.198 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:43:23.199 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:43:23.199 INFO:tasks.workunit.client.0.smithi172.stdout:3 2024-05-08T16:43:23.199 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-05-08T16:43:23.199 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:43:23.199 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-08T16:43:23.748 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:43:24.750 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:43:24.750 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:43:24.751 INFO:tasks.workunit.client.0.smithi172.stdout:4 2024-05-08T16:43:24.751 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-05-08T16:43:24.751 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:43:24.751 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-08T16:43:25.656 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:43:26.657 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:43:26.657 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:43:26.657 INFO:tasks.workunit.client.0.smithi172.stdout:5 2024-05-08T16:43:26.658 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-05-08T16:43:26.658 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:43:26.658 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-08T16:43:27.265 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:43:28.266 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:43:28.267 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:43:28.267 INFO:tasks.workunit.client.0.smithi172.stdout:6 2024-05-08T16:43:28.268 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-05-08T16:43:28.268 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:43:28.268 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-08T16:43:28.970 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:43:29.972 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:43:29.973 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:43:29.973 INFO:tasks.workunit.client.0.smithi172.stdout:7 2024-05-08T16:43:29.973 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-05-08T16:43:29.974 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:43:29.974 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-08T16:43:30.559 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:43:31.560 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:43:31.561 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:43:31.561 INFO:tasks.workunit.client.0.smithi172.stdout:8 2024-05-08T16:43:31.561 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-05-08T16:43:31.561 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:43:31.562 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-08T16:43:32.191 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:43:33.193 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:43:33.193 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:43:33.193 INFO:tasks.workunit.client.0.smithi172.stdout:9 2024-05-08T16:43:33.194 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-05-08T16:43:33.194 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:43:33.194 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-08T16:43:33.931 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:43:34.933 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:43:34.933 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:43:34.933 INFO:tasks.workunit.client.0.smithi172.stdout:10 2024-05-08T16:43:34.934 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-05-08T16:43:34.934 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:43:34.934 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-08T16:43:35.404 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:43:36.406 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:43:36.406 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:43:36.406 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-05-08T16:43:36.407 INFO:tasks.workunit.client.0.smithi172.stdout:11 2024-05-08T16:43:36.410 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:43:36.411 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-08T16:43:36.940 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:43:37.942 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:43:37.942 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:43:37.943 INFO:tasks.workunit.client.0.smithi172.stdout:12 2024-05-08T16:43:37.943 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-05-08T16:43:37.943 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:43:37.943 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-08T16:43:38.506 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:43:39.508 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:43:39.508 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:43:39.508 INFO:tasks.workunit.client.0.smithi172.stdout:13 2024-05-08T16:43:39.509 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-05-08T16:43:39.509 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:43:39.509 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-08T16:43:40.312 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:43:41.313 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:43:41.314 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:43:41.314 INFO:tasks.workunit.client.0.smithi172.stdout:14 2024-05-08T16:43:41.314 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-05-08T16:43:41.314 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:43:41.315 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-08T16:43:41.973 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:43:42.975 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:43:42.975 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:43:42.975 INFO:tasks.workunit.client.0.smithi172.stdout:15 2024-05-08T16:43:42.976 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-05-08T16:43:42.976 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:43:42.976 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-08T16:43:44.319 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:43:45.321 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:43:45.321 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:43:45.321 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-05-08T16:43:45.322 INFO:tasks.workunit.client.0.smithi172.stdout:16 2024-05-08T16:43:45.322 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:43:45.322 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-08T16:43:46.192 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:43:47.194 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:43:47.194 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:43:47.194 INFO:tasks.workunit.client.0.smithi172.stdout:17 2024-05-08T16:43:47.195 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-05-08T16:43:47.195 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:43:47.195 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-08T16:43:48.102 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:43:49.104 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:43:49.104 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:43:49.105 INFO:tasks.workunit.client.0.smithi172.stdout:18 2024-05-08T16:43:49.105 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-05-08T16:43:49.105 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:43:49.105 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-08T16:43:49.724 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:43:50.726 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:43:50.726 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:43:50.727 INFO:tasks.workunit.client.0.smithi172.stdout:19 2024-05-08T16:43:50.728 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-05-08T16:43:50.728 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:43:50.728 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-08T16:43:51.308 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:43:52.309 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:43:52.309 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:43:52.310 INFO:tasks.workunit.client.0.smithi172.stdout:20 2024-05-08T16:43:52.310 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-05-08T16:43:52.310 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:43:52.310 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-08T16:43:52.900 INFO:tasks.workunit.client.0.smithi172.stdout:osd.4 up in weight 1 up_from 39 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/923928435,v1:127.0.0.1:6835/923928435] [v2:127.0.0.1:6836/923928435,v1:127.0.0.1:6837/923928435] exists,up 7ed0bd11-b5df-4978-bf71-408483e65430 2024-05-08T16:43:52.905 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-08T16:43:52.905 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-08T16:43:52.905 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-08T16:43:52.905 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-05-08T16:43:52.905 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 5 2024-05-08T16:43:52.905 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-05-08T16:43:52.905 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-05-08T16:43:52.905 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=5 2024-05-08T16:43:52.905 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-05-08T16:43:52.906 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/5 2024-05-08T16:43:52.906 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=4457b0f1-028b-482b-8ee3-7ee05861e6cc --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 ' 2024-05-08T16:43:52.906 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-08T16:43:52.906 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-05-08T16:43:52.906 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-08T16:43:52.906 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/5' 2024-05-08T16:43:52.906 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/5/journal' 2024-05-08T16:43:52.906 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-05-08T16:43:52.906 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-05-08T16:43:52.906 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-05-08T16:43:52.906 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-05-08T16:43:52.907 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-08T16:43:52.907 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-08T16:43:52.907 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-08T16:43:52.907 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T16:43:52.907 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81110 2024-05-08T16:43:52.907 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.81110/$cluster-$name.asok' 2024-05-08T16:43:52.907 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.81110/$cluster-$name.asok' 2024-05-08T16:43:52.907 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-05-08T16:43:52.907 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-05-08T16:43:52.907 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-05-08T16:43:52.907 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-05-08T16:43:52.907 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-05-08T16:43:52.907 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-08T16:43:52.908 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-08T16:43:52.908 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-08T16:43:52.908 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-08T16:43:52.908 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-05-08T16:43:52.908 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-05-08T16:43:52.908 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/5 2024-05-08T16:43:52.910 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-05-08T16:43:52.910 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=f23e7d72-6da9-4b5b-88ae-841e0f8c8c29 2024-05-08T16:43:52.910 INFO:tasks.workunit.client.0.smithi172.stdout:add osd5 f23e7d72-6da9-4b5b-88ae-841e0f8c8c29 2024-05-08T16:43:52.911 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd5 f23e7d72-6da9-4b5b-88ae-841e0f8c8c29' 2024-05-08T16:43:52.911 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-05-08T16:43:52.935 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDIqztmPs0EOBAAkT0Tn+fHA7N9OKVtUAPQGg== 2024-05-08T16:43:52.935 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDIqztmPs0EOBAAkT0Tn+fHA7N9OKVtUAPQGg=="}' 2024-05-08T16:43:52.935 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new f23e7d72-6da9-4b5b-88ae-841e0f8c8c29 -i td/test-erasure-eio/5/new.json 2024-05-08T16:43:54.254 INFO:tasks.workunit.client.0.smithi172.stdout:5 2024-05-08T16:43:54.274 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/5/new.json 2024-05-08T16:43:54.276 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 5 --fsid=4457b0f1-028b-482b-8ee3-7ee05861e6cc --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/5 --osd-journal=td/test-erasure-eio/5/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81110/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDIqztmPs0EOBAAkT0Tn+fHA7N9OKVtUAPQGg== --osd-uuid f23e7d72-6da9-4b5b-88ae-841e0f8c8c29 2024-05-08T16:43:54.315 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:43:54.319+0000 7f6d218107c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:43:54.407 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:43:54.411+0000 7f6d218107c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:43:54.409 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:43:54.415+0000 7f6d218107c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:43:54.409 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:43:54.415+0000 7f6d218107c0 -1 bluestore(td/test-erasure-eio/5/block) _read_bdev_label failed to open td/test-erasure-eio/5/block: (2) No such file or directory 2024-05-08T16:43:54.409 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:43:54.415+0000 7f6d218107c0 -1 bluestore(td/test-erasure-eio/5/block) _read_bdev_label failed to open td/test-erasure-eio/5/block: (2) No such file or directory 2024-05-08T16:43:54.409 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:43:54.415+0000 7f6d218107c0 -1 bluestore(td/test-erasure-eio/5/block) _read_bdev_label failed to open td/test-erasure-eio/5/block: (2) No such file or directory 2024-05-08T16:43:54.572 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:43:54.575+0000 7f6d218107c0 -1 bluestore(td/test-erasure-eio/5) _read_fsid unparsable uuid 2024-05-08T16:44:02.703 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/5/keyring 2024-05-08T16:44:02.703 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-05-08T16:44:02.705 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd5 key to auth repository 2024-05-08T16:44:02.705 INFO:tasks.workunit.client.0.smithi172.stdout:adding osd5 key to auth repository 2024-05-08T16:44:02.705 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/5/keyring auth add osd.5 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-05-08T16:44:03.282 INFO:tasks.workunit.client.0.smithi172.stdout:start osd.5 2024-05-08T16:44:03.283 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.5 2024-05-08T16:44:03.283 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 5 --fsid=4457b0f1-028b-482b-8ee3-7ee05861e6cc --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/5 --osd-journal=td/test-erasure-eio/5/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81110/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-05-08T16:44:03.284 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-05-08T16:44:03.284 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-05-08T16:44:03.284 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-05-08T16:44:03.319 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:44:03.323+0000 7f29a82f27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:44:03.362 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:44:03.367+0000 7f29a82f27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:44:03.364 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:44:03.367+0000 7f29a82f27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:44:03.795 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 5 2024-05-08T16:44:03.795 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-08T16:44:03.795 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=5 2024-05-08T16:44:03.795 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-08T16:44:03.795 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-08T16:44:03.795 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:44:03.796 INFO:tasks.workunit.client.0.smithi172.stdout:0 2024-05-08T16:44:03.796 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-08T16:44:03.796 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:44:03.796 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-05-08T16:44:03.952 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:44:03.955+0000 7f29a82f27c0 -1 Falling back to public interface 2024-05-08T16:44:04.311 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:44:05.313 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:44:05.313 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:44:05.314 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-08T16:44:05.314 INFO:tasks.workunit.client.0.smithi172.stdout:1 2024-05-08T16:44:05.314 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:44:05.314 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-05-08T16:44:05.751 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:44:06.646 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:44:06.651+0000 7f29a82f27c0 -1 osd.5 0 log_to_monitors true 2024-05-08T16:44:06.752 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:44:06.753 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:44:06.753 INFO:tasks.workunit.client.0.smithi172.stdout:2 2024-05-08T16:44:06.753 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-08T16:44:06.754 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:44:06.754 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-05-08T16:44:07.282 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:44:08.284 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:44:08.284 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:44:08.285 INFO:tasks.workunit.client.0.smithi172.stdout:3 2024-05-08T16:44:08.285 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-05-08T16:44:08.285 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:44:08.285 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-05-08T16:44:09.113 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:44:10.114 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:44:10.115 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:44:10.115 INFO:tasks.workunit.client.0.smithi172.stdout:4 2024-05-08T16:44:10.115 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-05-08T16:44:10.115 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:44:10.116 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-05-08T16:44:10.766 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:44:11.768 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:44:11.768 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:44:11.768 INFO:tasks.workunit.client.0.smithi172.stdout:5 2024-05-08T16:44:11.769 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-05-08T16:44:11.769 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:44:11.769 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-05-08T16:44:12.412 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:44:13.414 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:44:13.414 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:44:13.414 INFO:tasks.workunit.client.0.smithi172.stdout:6 2024-05-08T16:44:13.415 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-05-08T16:44:13.415 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:44:13.415 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-05-08T16:44:14.331 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:44:15.333 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:44:15.333 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:44:15.333 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-05-08T16:44:15.333 INFO:tasks.workunit.client.0.smithi172.stdout:7 2024-05-08T16:44:15.334 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:44:15.334 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-05-08T16:44:15.957 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:44:16.958 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:44:16.959 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:44:16.959 INFO:tasks.workunit.client.0.smithi172.stdout:8 2024-05-08T16:44:16.959 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-05-08T16:44:16.959 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:44:16.959 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-05-08T16:44:17.437 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:44:18.439 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:44:18.439 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:44:18.439 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-05-08T16:44:18.440 INFO:tasks.workunit.client.0.smithi172.stdout:9 2024-05-08T16:44:18.440 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:44:18.440 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-05-08T16:44:19.106 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:44:20.107 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:44:20.108 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:44:20.108 INFO:tasks.workunit.client.0.smithi172.stdout:10 2024-05-08T16:44:20.108 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-05-08T16:44:20.108 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:44:20.109 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-05-08T16:44:20.774 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:44:21.776 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:44:21.776 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:44:21.776 INFO:tasks.workunit.client.0.smithi172.stdout:11 2024-05-08T16:44:21.777 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-05-08T16:44:21.777 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:44:21.777 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-05-08T16:44:22.367 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:44:23.370 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:44:23.370 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:44:23.370 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-05-08T16:44:23.370 INFO:tasks.workunit.client.0.smithi172.stdout:12 2024-05-08T16:44:23.371 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:44:23.371 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-05-08T16:44:24.037 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:44:25.038 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:44:25.038 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:44:25.039 INFO:tasks.workunit.client.0.smithi172.stdout:13 2024-05-08T16:44:25.039 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-05-08T16:44:25.039 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:44:25.039 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-05-08T16:44:25.971 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:44:26.973 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:44:26.973 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:44:26.973 INFO:tasks.workunit.client.0.smithi172.stdout:14 2024-05-08T16:44:26.974 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-05-08T16:44:26.974 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:44:26.974 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-05-08T16:44:27.585 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:44:28.582 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:44:28.582 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:44:28.582 INFO:tasks.workunit.client.0.smithi172.stdout:15 2024-05-08T16:44:28.583 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-05-08T16:44:28.583 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:44:28.583 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-05-08T16:44:29.964 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:44:30.966 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:44:30.966 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:44:30.966 INFO:tasks.workunit.client.0.smithi172.stdout:16 2024-05-08T16:44:30.967 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-05-08T16:44:30.967 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:44:30.967 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-05-08T16:44:31.771 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:44:32.773 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:44:32.773 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:44:32.773 INFO:tasks.workunit.client.0.smithi172.stdout:17 2024-05-08T16:44:32.774 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-05-08T16:44:32.774 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:44:32.774 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-05-08T16:44:34.168 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:44:35.170 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:44:35.171 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:44:35.171 INFO:tasks.workunit.client.0.smithi172.stdout:18 2024-05-08T16:44:35.172 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-05-08T16:44:35.172 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:44:35.172 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-05-08T16:44:36.223 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:44:37.225 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:44:37.226 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:44:37.226 INFO:tasks.workunit.client.0.smithi172.stdout:19 2024-05-08T16:44:37.226 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-05-08T16:44:37.227 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:44:37.227 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-05-08T16:44:37.646 INFO:tasks.workunit.client.0.smithi172.stdout:osd.5 up in weight 1 up_from 49 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6842/2068815838,v1:127.0.0.1:6843/2068815838] [v2:127.0.0.1:6844/2068815838,v1:127.0.0.1:6845/2068815838] exists,up f23e7d72-6da9-4b5b-88ae-841e0f8c8c29 2024-05-08T16:44:37.651 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-08T16:44:37.651 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-08T16:44:37.651 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-08T16:44:37.651 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-05-08T16:44:37.651 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 6 2024-05-08T16:44:37.651 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-05-08T16:44:37.651 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-05-08T16:44:37.651 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=6 2024-05-08T16:44:37.651 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-05-08T16:44:37.651 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/6 2024-05-08T16:44:37.652 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=4457b0f1-028b-482b-8ee3-7ee05861e6cc --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 ' 2024-05-08T16:44:37.652 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-08T16:44:37.652 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-05-08T16:44:37.652 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-08T16:44:37.652 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/6' 2024-05-08T16:44:37.652 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/6/journal' 2024-05-08T16:44:37.652 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-05-08T16:44:37.652 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-05-08T16:44:37.652 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-05-08T16:44:37.652 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-05-08T16:44:37.653 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-08T16:44:37.653 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-08T16:44:37.653 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-08T16:44:37.653 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T16:44:37.653 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81110 2024-05-08T16:44:37.653 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.81110/$cluster-$name.asok' 2024-05-08T16:44:37.653 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.81110/$cluster-$name.asok' 2024-05-08T16:44:37.653 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-05-08T16:44:37.653 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-05-08T16:44:37.653 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-05-08T16:44:37.653 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-05-08T16:44:37.653 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-05-08T16:44:37.654 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-08T16:44:37.654 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-08T16:44:37.654 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-08T16:44:37.654 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-08T16:44:37.654 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-05-08T16:44:37.654 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-05-08T16:44:37.654 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/6 2024-05-08T16:44:37.654 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-05-08T16:44:37.655 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=dd7103e9-87cb-4946-bcb6-a31ce35389a3 2024-05-08T16:44:37.655 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd6 dd7103e9-87cb-4946-bcb6-a31ce35389a3' 2024-05-08T16:44:37.655 INFO:tasks.workunit.client.0.smithi172.stdout:add osd6 dd7103e9-87cb-4946-bcb6-a31ce35389a3 2024-05-08T16:44:37.655 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-05-08T16:44:37.671 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQD1qztm4ONcKBAAoMXyuxCo1XoW612G0idTIA== 2024-05-08T16:44:37.671 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQD1qztm4ONcKBAAoMXyuxCo1XoW612G0idTIA=="}' 2024-05-08T16:44:37.671 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new dd7103e9-87cb-4946-bcb6-a31ce35389a3 -i td/test-erasure-eio/6/new.json 2024-05-08T16:44:38.608 INFO:tasks.workunit.client.0.smithi172.stdout:6 2024-05-08T16:44:38.630 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/6/new.json 2024-05-08T16:44:38.632 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 6 --fsid=4457b0f1-028b-482b-8ee3-7ee05861e6cc --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/6 --osd-journal=td/test-erasure-eio/6/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81110/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQD1qztm4ONcKBAAoMXyuxCo1XoW612G0idTIA== --osd-uuid dd7103e9-87cb-4946-bcb6-a31ce35389a3 2024-05-08T16:44:38.672 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:44:38.674+0000 7f63519b47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:44:38.689 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:44:38.690+0000 7f63519b47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:44:38.690 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:44:38.694+0000 7f63519b47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:44:38.690 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:44:38.694+0000 7f63519b47c0 -1 bluestore(td/test-erasure-eio/6/block) _read_bdev_label failed to open td/test-erasure-eio/6/block: (2) No such file or directory 2024-05-08T16:44:38.690 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:44:38.694+0000 7f63519b47c0 -1 bluestore(td/test-erasure-eio/6/block) _read_bdev_label failed to open td/test-erasure-eio/6/block: (2) No such file or directory 2024-05-08T16:44:38.690 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:44:38.694+0000 7f63519b47c0 -1 bluestore(td/test-erasure-eio/6/block) _read_bdev_label failed to open td/test-erasure-eio/6/block: (2) No such file or directory 2024-05-08T16:44:38.794 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:44:38.798+0000 7f63519b47c0 -1 bluestore(td/test-erasure-eio/6) _read_fsid unparsable uuid 2024-05-08T16:44:47.486 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/6/keyring 2024-05-08T16:44:47.486 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-05-08T16:44:47.487 INFO:tasks.workunit.client.0.smithi172.stdout:adding osd6 key to auth repository 2024-05-08T16:44:47.488 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd6 key to auth repository 2024-05-08T16:44:47.488 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/6/keyring auth add osd.6 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-05-08T16:44:48.054 INFO:tasks.workunit.client.0.smithi172.stdout:start osd.6 2024-05-08T16:44:48.055 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.6 2024-05-08T16:44:48.055 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-05-08T16:44:48.055 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-05-08T16:44:48.055 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 6 --fsid=4457b0f1-028b-482b-8ee3-7ee05861e6cc --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/6 --osd-journal=td/test-erasure-eio/6/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81110/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-05-08T16:44:48.055 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-05-08T16:44:48.091 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:44:48.094+0000 7f01533997c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:44:48.093 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:44:48.098+0000 7f01533997c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:44:48.094 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:44:48.098+0000 7f01533997c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:44:48.491 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 6 2024-05-08T16:44:48.491 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-08T16:44:48.492 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=6 2024-05-08T16:44:48.492 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-08T16:44:48.492 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-08T16:44:48.492 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:44:48.492 INFO:tasks.workunit.client.0.smithi172.stdout:0 2024-05-08T16:44:48.492 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-08T16:44:48.493 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:44:48.493 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-05-08T16:44:48.666 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:44:48.670+0000 7f01533997c0 -1 Falling back to public interface 2024-05-08T16:44:48.925 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:44:49.926 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:44:49.926 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:44:49.927 INFO:tasks.workunit.client.0.smithi172.stdout:1 2024-05-08T16:44:49.927 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-08T16:44:49.927 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:44:49.927 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-05-08T16:44:50.377 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:44:51.330 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:44:51.334+0000 7f01533997c0 -1 osd.6 0 log_to_monitors true 2024-05-08T16:44:51.380 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:44:51.380 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:44:51.380 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-08T16:44:51.380 INFO:tasks.workunit.client.0.smithi172.stdout:2 2024-05-08T16:44:51.381 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:44:51.381 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-05-08T16:44:52.243 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:44:53.245 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:44:53.245 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:44:53.245 INFO:tasks.workunit.client.0.smithi172.stdout:3 2024-05-08T16:44:53.246 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-05-08T16:44:53.246 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:44:53.246 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-05-08T16:44:54.164 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:44:55.165 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:44:55.165 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:44:55.165 INFO:tasks.workunit.client.0.smithi172.stdout:4 2024-05-08T16:44:55.166 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-05-08T16:44:55.166 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:44:55.166 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-05-08T16:44:55.887 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:44:56.889 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:44:56.889 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:44:56.889 INFO:tasks.workunit.client.0.smithi172.stdout:5 2024-05-08T16:44:56.890 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-05-08T16:44:56.890 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:44:56.890 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-05-08T16:44:57.411 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:44:58.413 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:44:58.413 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:44:58.414 INFO:tasks.workunit.client.0.smithi172.stdout:6 2024-05-08T16:44:58.414 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-05-08T16:44:58.414 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:44:58.414 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-05-08T16:44:58.937 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:44:59.939 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:44:59.939 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:44:59.939 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-05-08T16:44:59.939 INFO:tasks.workunit.client.0.smithi172.stdout:7 2024-05-08T16:44:59.940 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:44:59.940 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-05-08T16:45:00.664 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:45:01.666 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:45:01.666 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:45:01.666 INFO:tasks.workunit.client.0.smithi172.stdout:8 2024-05-08T16:45:01.667 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-05-08T16:45:01.667 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:45:01.667 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-05-08T16:45:02.276 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:45:03.278 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:45:03.278 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:45:03.278 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-05-08T16:45:03.279 INFO:tasks.workunit.client.0.smithi172.stdout:9 2024-05-08T16:45:03.279 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:45:03.279 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-05-08T16:45:03.852 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:45:04.854 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:45:04.854 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:45:04.854 INFO:tasks.workunit.client.0.smithi172.stdout:10 2024-05-08T16:45:04.855 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-05-08T16:45:04.855 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:45:04.855 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-05-08T16:45:05.486 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:45:06.488 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:45:06.488 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:45:06.489 INFO:tasks.workunit.client.0.smithi172.stdout:11 2024-05-08T16:45:06.489 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-05-08T16:45:06.489 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:45:06.489 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-05-08T16:45:06.975 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:45:07.976 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:45:07.976 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:45:07.977 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-05-08T16:45:07.977 INFO:tasks.workunit.client.0.smithi172.stdout:12 2024-05-08T16:45:07.977 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:45:07.977 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-05-08T16:45:08.609 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:45:09.611 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:45:09.611 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:45:09.611 INFO:tasks.workunit.client.0.smithi172.stdout:13 2024-05-08T16:45:09.612 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-05-08T16:45:09.612 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:45:09.612 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-05-08T16:45:10.751 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:45:11.753 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:45:11.753 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:45:11.753 INFO:tasks.workunit.client.0.smithi172.stdout:14 2024-05-08T16:45:11.754 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-05-08T16:45:11.754 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:45:11.754 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-05-08T16:45:12.922 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:45:13.924 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:45:13.924 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:45:13.924 INFO:tasks.workunit.client.0.smithi172.stdout:15 2024-05-08T16:45:13.925 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-05-08T16:45:13.925 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:45:13.925 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-05-08T16:45:14.870 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:45:15.872 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:45:15.872 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:45:15.872 INFO:tasks.workunit.client.0.smithi172.stdout:16 2024-05-08T16:45:15.873 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-05-08T16:45:15.873 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:45:15.873 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-05-08T16:45:16.602 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:45:17.604 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:45:17.604 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:45:17.604 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-05-08T16:45:17.604 INFO:tasks.workunit.client.0.smithi172.stdout:17 2024-05-08T16:45:17.604 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:45:17.605 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-05-08T16:45:18.555 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:45:19.557 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:45:19.557 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:45:19.557 INFO:tasks.workunit.client.0.smithi172.stdout:18 2024-05-08T16:45:19.558 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-05-08T16:45:19.558 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:45:19.558 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-05-08T16:45:20.191 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:45:21.193 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:45:21.193 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:45:21.194 INFO:tasks.workunit.client.0.smithi172.stdout:19 2024-05-08T16:45:21.195 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-05-08T16:45:21.195 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:45:21.195 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-05-08T16:45:21.806 INFO:tasks.workunit.client.0.smithi172.stdout:osd.6 up in weight 1 up_from 60 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6850/2947595983,v1:127.0.0.1:6851/2947595983] [v2:127.0.0.1:6852/2947595983,v1:127.0.0.1:6853/2947595983] exists,up dd7103e9-87cb-4946-bcb6-a31ce35389a3 2024-05-08T16:45:21.807 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-08T16:45:21.807 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-08T16:45:21.807 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-08T16:45:21.807 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: get_asok_path osd.0 2024-05-08T16:45:21.808 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-05-08T16:45:21.808 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-05-08T16:45:21.809 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-08T16:45:21.809 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T16:45:21.809 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81110 2024-05-08T16:45:21.809 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.81110/ceph-osd.0.asok 2024-05-08T16:45:21.809 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: CEPH_ARGS= 2024-05-08T16:45:21.809 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: ceph --admin-daemon /tmp/ceph-asok.81110/ceph-osd.0.asok log flush 2024-05-08T16:45:21.922 INFO:tasks.workunit.client.0.smithi172.stdout:{} 2024-05-08T16:45:21.935 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:56: setup_osds: grep 'load: jerasure.*lrc' td/test-erasure-eio/osd.0.log 2024-05-08T16:45:21.943 INFO:tasks.workunit.client.0.smithi172.stdout:2024-05-08T16:40:27.227+0000 7f08497937c0 0 load: jerasure load: lrc 2024-05-08T16:45:21.944 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:486: TEST_ec_recovery_multiple_objects_eio: CEPH_ARGS='--fsid=4457b0f1-028b-482b-8ee3-7ee05861e6cc --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-05-08T16:45:21.944 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:488: TEST_ec_recovery_multiple_objects_eio: local poolname=pool-jerasure 2024-05-08T16:45:21.944 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:489: TEST_ec_recovery_multiple_objects_eio: create_erasure_coded_pool pool-jerasure 3 2 2024-05-08T16:45:21.944 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:67: create_erasure_coded_pool: local poolname=pool-jerasure 2024-05-08T16:45:21.944 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:68: create_erasure_coded_pool: shift 2024-05-08T16:45:21.944 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:69: create_erasure_coded_pool: local k=3 2024-05-08T16:45:21.944 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:70: create_erasure_coded_pool: shift 2024-05-08T16:45:21.945 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:71: create_erasure_coded_pool: local m=2 2024-05-08T16:45:21.945 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:72: create_erasure_coded_pool: shift 2024-05-08T16:45:21.945 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:74: create_erasure_coded_pool: ceph osd erasure-code-profile set myprofile plugin=jerasure k=3 m=2 crush-failure-domain=osd 2024-05-08T16:45:23.299 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:78: create_erasure_coded_pool: create_pool pool-jerasure 1 1 erasure myprofile 2024-05-08T16:45:23.299 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create pool-jerasure 1 1 erasure myprofile 2024-05-08T16:45:24.735 INFO:tasks.workunit.client.0.smithi172.stderr:pool 'pool-jerasure' already exists 2024-05-08T16:45:24.753 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-05-08T16:45:25.755 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:80: create_erasure_coded_pool: wait_for_clean 2024-05-08T16:45:25.755 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-05-08T16:45:25.755 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-05-08T16:45:25.755 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-05-08T16:45:25.755 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-05-08T16:45:25.756 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-05-08T16:45:25.756 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-05-08T16:45:25.757 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-05-08T16:45:25.757 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-05-08T16:45:25.757 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-05-08T16:45:25.883 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-05-08T16:45:25.883 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-05-08T16:45:25.883 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-05-08T16:45:25.883 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-05-08T16:45:25.883 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-05-08T16:45:25.884 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-05-08T16:45:26.319 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-05-08T16:45:26.319 INFO:tasks.workunit.client.0.smithi172.stderr:1 2024-05-08T16:45:26.319 INFO:tasks.workunit.client.0.smithi172.stderr:2 2024-05-08T16:45:26.319 INFO:tasks.workunit.client.0.smithi172.stderr:3 2024-05-08T16:45:26.319 INFO:tasks.workunit.client.0.smithi172.stderr:4 2024-05-08T16:45:26.319 INFO:tasks.workunit.client.0.smithi172.stderr:5 2024-05-08T16:45:26.319 INFO:tasks.workunit.client.0.smithi172.stderr:6' 2024-05-08T16:45:26.319 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-05-08T16:45:26.319 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T16:45:26.320 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-05-08T16:45:26.592 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=25769803838 2024-05-08T16:45:26.593 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 25769803838 2024-05-08T16:45:26.593 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803838' 2024-05-08T16:45:26.593 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T16:45:26.593 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-05-08T16:45:26.830 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=55834574901 2024-05-08T16:45:26.830 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 55834574901 2024-05-08T16:45:26.830 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803838 1-55834574901' 2024-05-08T16:45:26.830 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T16:45:26.830 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-05-08T16:45:27.129 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=81604378665 2024-05-08T16:45:27.130 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 81604378665 2024-05-08T16:45:27.130 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803838 1-55834574901 2-81604378665' 2024-05-08T16:45:27.130 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T16:45:27.130 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-05-08T16:45:27.312 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=120259084323 2024-05-08T16:45:27.312 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 120259084323 2024-05-08T16:45:27.313 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803838 1-55834574901 2-81604378665 3-120259084323' 2024-05-08T16:45:27.313 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T16:45:27.313 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-05-08T16:45:27.487 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=167503724566 2024-05-08T16:45:27.487 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 167503724566 2024-05-08T16:45:27.487 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803838 1-55834574901 2-81604378665 3-120259084323 4-167503724566' 2024-05-08T16:45:27.487 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T16:45:27.487 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-05-08T16:45:27.643 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=210453397521 2024-05-08T16:45:27.643 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 210453397521 2024-05-08T16:45:27.643 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803838 1-55834574901 2-81604378665 3-120259084323 4-167503724566 5-210453397521' 2024-05-08T16:45:27.643 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T16:45:27.643 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-05-08T16:45:27.799 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=257698037768 2024-05-08T16:45:27.799 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 257698037768 2024-05-08T16:45:27.799 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803838 1-55834574901 2-81604378665 3-120259084323 4-167503724566 5-210453397521 6-257698037768' 2024-05-08T16:45:27.799 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-08T16:45:27.800 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-25769803838 2024-05-08T16:45:27.800 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-08T16:45:27.800 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-05-08T16:45:27.801 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-25769803838 2024-05-08T16:45:27.801 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-08T16:45:27.804 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=25769803838 2024-05-08T16:45:27.804 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 25769803838' 2024-05-08T16:45:27.804 INFO:tasks.workunit.client.0.smithi172.stdout:waiting osd.0 seq 25769803838 2024-05-08T16:45:27.805 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-05-08T16:45:28.209 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 25769803838 -lt 25769803838 2024-05-08T16:45:28.209 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-08T16:45:28.211 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-55834574901 2024-05-08T16:45:28.211 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-08T16:45:28.212 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-05-08T16:45:28.213 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-55834574901 2024-05-08T16:45:28.213 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-08T16:45:28.214 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=55834574901 2024-05-08T16:45:28.214 INFO:tasks.workunit.client.0.smithi172.stdout:waiting osd.1 seq 55834574901 2024-05-08T16:45:28.214 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 55834574901' 2024-05-08T16:45:28.214 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-05-08T16:45:28.645 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 55834574901 -lt 55834574901 2024-05-08T16:45:28.645 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-08T16:45:28.645 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-81604378665 2024-05-08T16:45:28.645 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-08T16:45:28.647 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-05-08T16:45:28.648 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-81604378665 2024-05-08T16:45:28.648 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-08T16:45:28.649 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=81604378665 2024-05-08T16:45:28.649 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 81604378665' 2024-05-08T16:45:28.649 INFO:tasks.workunit.client.0.smithi172.stdout:waiting osd.2 seq 81604378665 2024-05-08T16:45:28.649 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-05-08T16:45:29.096 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 81604378665 -lt 81604378665 2024-05-08T16:45:29.096 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-08T16:45:29.097 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-120259084323 2024-05-08T16:45:29.097 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-08T16:45:29.098 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-05-08T16:45:29.099 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-120259084323 2024-05-08T16:45:29.099 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-08T16:45:29.100 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=120259084323 2024-05-08T16:45:29.101 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 120259084323' 2024-05-08T16:45:29.101 INFO:tasks.workunit.client.0.smithi172.stdout:waiting osd.3 seq 120259084323 2024-05-08T16:45:29.101 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-05-08T16:45:29.582 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 120259084323 -lt 120259084323 2024-05-08T16:45:29.582 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-08T16:45:29.582 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-167503724566 2024-05-08T16:45:29.583 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-08T16:45:29.584 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-05-08T16:45:29.585 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-167503724566 2024-05-08T16:45:29.585 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-08T16:45:29.586 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=167503724566 2024-05-08T16:45:29.587 INFO:tasks.workunit.client.0.smithi172.stdout:waiting osd.4 seq 167503724566 2024-05-08T16:45:29.587 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 167503724566' 2024-05-08T16:45:29.587 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-05-08T16:45:29.973 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 167503724567 -lt 167503724566 2024-05-08T16:45:29.973 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-08T16:45:29.974 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-210453397521 2024-05-08T16:45:29.974 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-08T16:45:29.975 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-05-08T16:45:29.976 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-210453397521 2024-05-08T16:45:29.976 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-08T16:45:29.977 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=210453397521 2024-05-08T16:45:29.977 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 210453397521' 2024-05-08T16:45:29.977 INFO:tasks.workunit.client.0.smithi172.stdout:waiting osd.5 seq 210453397521 2024-05-08T16:45:29.977 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-05-08T16:45:30.412 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 210453397521 -lt 210453397521 2024-05-08T16:45:30.412 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-08T16:45:30.412 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-257698037768 2024-05-08T16:45:30.412 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-08T16:45:30.414 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-05-08T16:45:30.415 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-257698037768 2024-05-08T16:45:30.415 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-08T16:45:30.416 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=257698037768 2024-05-08T16:45:30.416 INFO:tasks.workunit.client.0.smithi172.stdout:waiting osd.6 seq 257698037768 2024-05-08T16:45:30.417 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 257698037768' 2024-05-08T16:45:30.417 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-05-08T16:45:30.852 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 257698037768 -lt 257698037768 2024-05-08T16:45:30.852 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-05-08T16:45:30.852 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-05-08T16:45:30.853 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-05-08T16:45:31.365 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-05-08T16:45:31.365 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-05-08T16:45:31.366 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-05-08T16:45:31.366 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-05-08T16:45:31.366 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-05-08T16:45:31.367 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-05-08T16:45:31.367 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-05-08T16:45:31.367 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-05-08T16:45:31.771 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-05-08T16:45:31.772 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-05-08T16:45:31.772 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-05-08T16:45:31.772 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-05-08T16:45:32.285 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-05-08T16:45:32.285 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-05-08T16:45:32.285 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-05-08T16:45:32.285 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:491: TEST_ec_recovery_multiple_objects_eio: rados_put td/test-erasure-eio pool-jerasure test1 2024-05-08T16:45:32.285 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:90: rados_put: local dir=td/test-erasure-eio 2024-05-08T16:45:32.285 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:91: rados_put: local poolname=pool-jerasure 2024-05-08T16:45:32.285 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:92: rados_put: local objname=test1 2024-05-08T16:45:32.285 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-05-08T16:45:32.285 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 AAA 2024-05-08T16:45:32.286 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-05-08T16:45:32.286 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 BBB 2024-05-08T16:45:32.286 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-05-08T16:45:32.286 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 CCCC 2024-05-08T16:45:32.286 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-05-08T16:45:32.286 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 DDDD 2024-05-08T16:45:32.286 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:100: rados_put: rados --pool pool-jerasure put test1 td/test-erasure-eio/ORIGINAL 2024-05-08T16:45:32.531 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:492: TEST_ec_recovery_multiple_objects_eio: rados_put td/test-erasure-eio pool-jerasure test2 2024-05-08T16:45:32.532 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:90: rados_put: local dir=td/test-erasure-eio 2024-05-08T16:45:32.532 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:91: rados_put: local poolname=pool-jerasure 2024-05-08T16:45:32.532 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:92: rados_put: local objname=test2 2024-05-08T16:45:32.532 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-05-08T16:45:32.532 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 AAA 2024-05-08T16:45:32.532 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-05-08T16:45:32.532 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 BBB 2024-05-08T16:45:32.532 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-05-08T16:45:32.532 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 CCCC 2024-05-08T16:45:32.532 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-05-08T16:45:32.532 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 DDDD 2024-05-08T16:45:32.532 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:100: rados_put: rados --pool pool-jerasure put test2 td/test-erasure-eio/ORIGINAL 2024-05-08T16:45:32.891 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:493: TEST_ec_recovery_multiple_objects_eio: rados_put td/test-erasure-eio pool-jerasure test3 2024-05-08T16:45:32.891 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:90: rados_put: local dir=td/test-erasure-eio 2024-05-08T16:45:32.891 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:91: rados_put: local poolname=pool-jerasure 2024-05-08T16:45:32.891 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:92: rados_put: local objname=test3 2024-05-08T16:45:32.922 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-05-08T16:45:32.922 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 AAA 2024-05-08T16:45:32.923 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-05-08T16:45:32.923 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 BBB 2024-05-08T16:45:32.923 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-05-08T16:45:32.923 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 CCCC 2024-05-08T16:45:32.923 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-05-08T16:45:32.923 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 DDDD 2024-05-08T16:45:32.923 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:100: rados_put: rados --pool pool-jerasure put test3 td/test-erasure-eio/ORIGINAL 2024-05-08T16:45:33.244 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:496: TEST_ec_recovery_multiple_objects_eio: inject_eio ec data pool-jerasure myobject td/test-erasure-eio 0 2024-05-08T16:45:33.244 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2394: inject_eio: local pooltype=ec 2024-05-08T16:45:33.244 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2395: inject_eio: shift 2024-05-08T16:45:33.245 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2396: inject_eio: local which=data 2024-05-08T16:45:33.245 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2397: inject_eio: shift 2024-05-08T16:45:33.245 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2398: inject_eio: local poolname=pool-jerasure 2024-05-08T16:45:33.245 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2399: inject_eio: shift 2024-05-08T16:45:33.245 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2400: inject_eio: local objname=myobject 2024-05-08T16:45:33.245 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2401: inject_eio: shift 2024-05-08T16:45:33.245 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2402: inject_eio: local dir=td/test-erasure-eio 2024-05-08T16:45:33.245 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2403: inject_eio: shift 2024-05-08T16:45:33.245 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2404: inject_eio: local shard_id=0 2024-05-08T16:45:33.245 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2405: inject_eio: shift 2024-05-08T16:45:33.245 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: get_osds pool-jerasure myobject 2024-05-08T16:45:33.245 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2024-05-08T16:45:33.245 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=myobject 2024-05-08T16:45:33.246 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure myobject 2024-05-08T16:45:33.246 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-05-08T16:45:33.671 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-05-08T16:45:33.671 INFO:tasks.workunit.client.0.smithi172.stderr:5 2024-05-08T16:45:33.671 INFO:tasks.workunit.client.0.smithi172.stderr:1 2024-05-08T16:45:33.671 INFO:tasks.workunit.client.0.smithi172.stderr:0 2024-05-08T16:45:33.671 INFO:tasks.workunit.client.0.smithi172.stderr:6' 2024-05-08T16:45:33.671 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 5 1 0 6 2024-05-08T16:45:33.671 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: initial_osds=('3' '5' '1' '0' '6') 2024-05-08T16:45:33.672 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: local -a initial_osds 2024-05-08T16:45:33.672 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2408: inject_eio: local osd_id=3 2024-05-08T16:45:33.672 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2409: inject_eio: '[' ec '!=' ec ']' 2024-05-08T16:45:33.672 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: cat td/test-erasure-eio/3/type 2024-05-08T16:45:33.672 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: type=bluestore 2024-05-08T16:45:33.673 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: set_config osd 3 bluestore_debug_inject_read_err true 2024-05-08T16:45:33.673 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-05-08T16:45:33.673 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-05-08T16:45:33.673 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-05-08T16:45:33.673 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-05-08T16:45:33.674 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-05-08T16:45:33.674 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-05-08T16:45:33.674 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-05-08T16:45:33.674 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-05-08T16:45:33.674 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-08T16:45:33.674 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T16:45:33.674 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81110 2024-05-08T16:45:33.675 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.81110/ceph-osd.3.asok 2024-05-08T16:45:33.675 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.81110/ceph-osd.3.asok config set bluestore_debug_inject_read_err true 2024-05-08T16:45:33.791 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-05-08T16:45:33.791 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local loop=0 2024-05-08T16:45:33.792 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: grep -q Invalid 2024-05-08T16:45:33.793 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: get_asok_path osd.3 2024-05-08T16:45:33.793 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-05-08T16:45:33.793 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-05-08T16:45:33.794 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-08T16:45:33.794 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T16:45:33.794 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81110 2024-05-08T16:45:33.794 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.81110/ceph-osd.3.asok 2024-05-08T16:45:33.794 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: CEPH_ARGS= 2024-05-08T16:45:33.794 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: ceph --admin-daemon /tmp/ceph-asok.81110/ceph-osd.3.asok injectdataerr pool-jerasure myobject 0 2024-05-08T16:45:33.913 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:497: TEST_ec_recovery_multiple_objects_eio: ceph osd out 0 2024-05-08T16:45:34.467 INFO:tasks.workunit.client.0.smithi172.stderr:osd.0 is already out. 2024-05-08T16:45:34.483 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:500: TEST_ec_recovery_multiple_objects_eio: wait_for_clean 2024-05-08T16:45:34.483 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-05-08T16:45:34.483 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-05-08T16:45:34.483 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-05-08T16:45:34.484 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-05-08T16:45:34.484 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-05-08T16:45:34.484 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-05-08T16:45:34.485 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-05-08T16:45:34.485 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-05-08T16:45:34.485 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-05-08T16:45:34.580 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-05-08T16:45:34.580 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-05-08T16:45:34.580 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-05-08T16:45:34.580 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-05-08T16:45:34.580 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-05-08T16:45:34.580 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-05-08T16:45:35.267 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-05-08T16:45:35.268 INFO:tasks.workunit.client.0.smithi172.stderr:1 2024-05-08T16:45:35.268 INFO:tasks.workunit.client.0.smithi172.stderr:2 2024-05-08T16:45:35.268 INFO:tasks.workunit.client.0.smithi172.stderr:3 2024-05-08T16:45:35.268 INFO:tasks.workunit.client.0.smithi172.stderr:4 2024-05-08T16:45:35.268 INFO:tasks.workunit.client.0.smithi172.stderr:5 2024-05-08T16:45:35.268 INFO:tasks.workunit.client.0.smithi172.stderr:6' 2024-05-08T16:45:35.268 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-05-08T16:45:35.268 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T16:45:35.268 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-05-08T16:45:35.581 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=25769803842 2024-05-08T16:45:35.581 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 25769803842 2024-05-08T16:45:35.581 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803842' 2024-05-08T16:45:35.581 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T16:45:35.582 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-05-08T16:45:35.851 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=55834574904 2024-05-08T16:45:35.852 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 55834574904 2024-05-08T16:45:35.852 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803842 1-55834574904' 2024-05-08T16:45:35.852 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T16:45:35.852 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-05-08T16:45:36.167 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=81604378669 2024-05-08T16:45:36.167 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 81604378669 2024-05-08T16:45:36.167 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803842 1-55834574904 2-81604378669' 2024-05-08T16:45:36.167 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T16:45:36.167 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-05-08T16:45:36.325 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=120259084327 2024-05-08T16:45:36.325 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 120259084327 2024-05-08T16:45:36.325 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803842 1-55834574904 2-81604378669 3-120259084327' 2024-05-08T16:45:36.325 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T16:45:36.325 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-05-08T16:45:36.617 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=167503724570 2024-05-08T16:45:36.618 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 167503724570 2024-05-08T16:45:36.618 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803842 1-55834574904 2-81604378669 3-120259084327 4-167503724570' 2024-05-08T16:45:36.618 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T16:45:36.618 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-05-08T16:45:36.773 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=210453397524 2024-05-08T16:45:36.773 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 210453397524 2024-05-08T16:45:36.773 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803842 1-55834574904 2-81604378669 3-120259084327 4-167503724570 5-210453397524' 2024-05-08T16:45:36.773 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T16:45:36.773 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-05-08T16:45:36.931 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=257698037771 2024-05-08T16:45:36.931 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 257698037771 2024-05-08T16:45:36.931 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803842 1-55834574904 2-81604378669 3-120259084327 4-167503724570 5-210453397524 6-257698037771' 2024-05-08T16:45:36.931 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-08T16:45:36.931 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-25769803842 2024-05-08T16:45:36.931 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-08T16:45:36.932 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-05-08T16:45:36.933 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-25769803842 2024-05-08T16:45:36.933 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-08T16:45:36.934 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=25769803842 2024-05-08T16:45:36.934 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 25769803842' 2024-05-08T16:45:36.934 INFO:tasks.workunit.client.0.smithi172.stdout:waiting osd.0 seq 25769803842 2024-05-08T16:45:36.934 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-05-08T16:45:37.393 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 0 -lt 25769803842 2024-05-08T16:45:37.393 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-05-08T16:45:38.394 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-05-08T16:45:38.395 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-05-08T16:45:38.824 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 25769803842 -lt 25769803842 2024-05-08T16:45:38.825 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-08T16:45:38.826 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-55834574904 2024-05-08T16:45:38.826 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-08T16:45:38.827 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-05-08T16:45:38.828 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-55834574904 2024-05-08T16:45:38.828 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-08T16:45:38.829 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=55834574904 2024-05-08T16:45:38.829 INFO:tasks.workunit.client.0.smithi172.stdout:waiting osd.1 seq 55834574904 2024-05-08T16:45:38.830 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 55834574904' 2024-05-08T16:45:38.830 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-05-08T16:45:39.296 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 55834574905 -lt 55834574904 2024-05-08T16:45:39.296 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-08T16:45:39.297 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-81604378669 2024-05-08T16:45:39.297 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-08T16:45:39.298 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-05-08T16:45:39.299 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-81604378669 2024-05-08T16:45:39.299 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-08T16:45:39.300 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=81604378669 2024-05-08T16:45:39.300 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 81604378669' 2024-05-08T16:45:39.300 INFO:tasks.workunit.client.0.smithi172.stdout:waiting osd.2 seq 81604378669 2024-05-08T16:45:39.300 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-05-08T16:45:39.738 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 81604378670 -lt 81604378669 2024-05-08T16:45:39.738 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-08T16:45:39.738 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-120259084327 2024-05-08T16:45:39.739 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-08T16:45:39.740 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-05-08T16:45:39.741 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-120259084327 2024-05-08T16:45:39.741 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-08T16:45:39.743 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=120259084327 2024-05-08T16:45:39.743 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 120259084327' 2024-05-08T16:45:39.743 INFO:tasks.workunit.client.0.smithi172.stdout:waiting osd.3 seq 120259084327 2024-05-08T16:45:39.744 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-05-08T16:45:40.295 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 120259084328 -lt 120259084327 2024-05-08T16:45:40.295 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-08T16:45:40.296 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-167503724570 2024-05-08T16:45:40.296 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-08T16:45:40.298 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-05-08T16:45:40.298 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-167503724570 2024-05-08T16:45:40.298 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-08T16:45:40.300 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=167503724570 2024-05-08T16:45:40.300 INFO:tasks.workunit.client.0.smithi172.stdout:waiting osd.4 seq 167503724570 2024-05-08T16:45:40.301 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 167503724570' 2024-05-08T16:45:40.301 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-05-08T16:45:40.687 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 167503724571 -lt 167503724570 2024-05-08T16:45:40.687 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-08T16:45:40.687 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-210453397524 2024-05-08T16:45:40.688 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-08T16:45:40.689 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-05-08T16:45:40.690 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-210453397524 2024-05-08T16:45:40.690 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-08T16:45:40.692 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=210453397524 2024-05-08T16:45:40.692 INFO:tasks.workunit.client.0.smithi172.stdout:waiting osd.5 seq 210453397524 2024-05-08T16:45:40.693 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 210453397524' 2024-05-08T16:45:40.693 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-05-08T16:45:41.120 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 210453397525 -lt 210453397524 2024-05-08T16:45:41.120 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-08T16:45:41.121 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-257698037771 2024-05-08T16:45:41.121 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-08T16:45:41.122 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-05-08T16:45:41.123 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-257698037771 2024-05-08T16:45:41.123 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-08T16:45:41.124 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=257698037771 2024-05-08T16:45:41.124 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 257698037771' 2024-05-08T16:45:41.124 INFO:tasks.workunit.client.0.smithi172.stdout:waiting osd.6 seq 257698037771 2024-05-08T16:45:41.125 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-05-08T16:45:41.539 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 257698037772 -lt 257698037771 2024-05-08T16:45:41.540 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-05-08T16:45:41.541 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-05-08T16:45:41.541 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-05-08T16:45:42.102 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-05-08T16:45:42.102 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-05-08T16:45:42.103 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-05-08T16:45:42.103 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-05-08T16:45:42.103 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-05-08T16:45:42.103 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-05-08T16:45:42.103 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-05-08T16:45:42.103 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-05-08T16:45:42.503 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-05-08T16:45:42.504 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-05-08T16:45:42.504 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-05-08T16:45:42.504 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-05-08T16:45:43.048 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-05-08T16:45:43.048 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-05-08T16:45:43.048 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-05-08T16:45:43.048 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:502: TEST_ec_recovery_multiple_objects_eio: rados_get td/test-erasure-eio pool-jerasure test1 2024-05-08T16:45:43.048 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2024-05-08T16:45:43.048 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2024-05-08T16:45:43.048 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=test1 2024-05-08T16:45:43.048 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=ok 2024-05-08T16:45:43.048 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' ok = fail ']' 2024-05-08T16:45:43.048 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:120: rados_get: rados --pool pool-jerasure get test1 td/test-erasure-eio/COPY 2024-05-08T16:45:43.232 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:121: rados_get: diff td/test-erasure-eio/ORIGINAL td/test-erasure-eio/COPY 2024-05-08T16:45:43.233 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:122: rados_get: rm td/test-erasure-eio/COPY 2024-05-08T16:45:43.234 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:503: TEST_ec_recovery_multiple_objects_eio: rados_get td/test-erasure-eio pool-jerasure test2 2024-05-08T16:45:43.234 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2024-05-08T16:45:43.234 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2024-05-08T16:45:43.235 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=test2 2024-05-08T16:45:43.235 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=ok 2024-05-08T16:45:43.235 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' ok = fail ']' 2024-05-08T16:45:43.235 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:120: rados_get: rados --pool pool-jerasure get test2 td/test-erasure-eio/COPY 2024-05-08T16:45:43.269 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:121: rados_get: diff td/test-erasure-eio/ORIGINAL td/test-erasure-eio/COPY 2024-05-08T16:45:43.270 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:122: rados_get: rm td/test-erasure-eio/COPY 2024-05-08T16:45:43.271 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:504: TEST_ec_recovery_multiple_objects_eio: rados_get td/test-erasure-eio pool-jerasure test3 2024-05-08T16:45:43.272 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2024-05-08T16:45:43.272 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2024-05-08T16:45:43.272 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=test3 2024-05-08T16:45:43.272 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=ok 2024-05-08T16:45:43.272 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' ok = fail ']' 2024-05-08T16:45:43.272 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:120: rados_get: rados --pool pool-jerasure get test3 td/test-erasure-eio/COPY 2024-05-08T16:45:43.301 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:121: rados_get: diff td/test-erasure-eio/ORIGINAL td/test-erasure-eio/COPY 2024-05-08T16:45:43.302 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:122: rados_get: rm td/test-erasure-eio/COPY 2024-05-08T16:45:43.303 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:506: TEST_ec_recovery_multiple_objects_eio: delete_erasure_coded_pool pool-jerasure 2024-05-08T16:45:43.303 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:84: delete_erasure_coded_pool: local poolname=pool-jerasure 2024-05-08T16:45:43.303 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:85: delete_erasure_coded_pool: ceph osd pool delete pool-jerasure pool-jerasure --yes-i-really-really-mean-it 2024-05-08T16:45:43.838 INFO:tasks.workunit.client.0.smithi172.stderr:pool 'pool-jerasure' does not exist 2024-05-08T16:45:43.854 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:86: delete_erasure_coded_pool: ceph osd erasure-code-profile rm myprofile 2024-05-08T16:45:44.870 INFO:tasks.workunit.client.0.smithi172.stderr:erasure-code-profile myprofile does not exist 2024-05-08T16:45:44.888 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:42: run: teardown td/test-erasure-eio 2024-05-08T16:45:44.889 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-eio 2024-05-08T16:45:44.889 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-05-08T16:45:44.889 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-eio KILL 2024-05-08T16:45:44.889 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-08T16:45:44.889 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-08T16:45:44.889 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-08T16:45:44.890 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-08T16:45:44.890 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-08T16:45:45.201 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-08T16:45:45.201 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-05-08T16:45:45.203 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-05-08T16:45:45.203 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-05-08T16:45:45.205 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-05-08T16:45:45.205 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-05-08T16:45:45.205 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-05-08T16:45:45.206 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-08T16:45:45.206 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-05-08T16:45:45.207 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-05-08T16:45:45.207 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-08T16:45:45.208 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-05-08T16:45:45.209 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-05-08T16:45:45.209 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-eio 2024-05-08T16:45:45.251 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-05-08T16:45:45.251 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T16:45:45.251 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81110 2024-05-08T16:45:45.251 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.81110 2024-05-08T16:45:45.252 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-05-08T16:45:45.253 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-05-08T16:45:45.253 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:32: run: for func in $funcs 2024-05-08T16:45:45.253 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:33: run: setup td/test-erasure-eio 2024-05-08T16:45:45.253 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/test-erasure-eio 2024-05-08T16:45:45.253 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/test-erasure-eio 2024-05-08T16:45:45.253 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-eio 2024-05-08T16:45:45.253 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-05-08T16:45:45.253 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-eio KILL 2024-05-08T16:45:45.254 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-08T16:45:45.254 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-08T16:45:45.254 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-08T16:45:45.254 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-08T16:45:45.254 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-08T16:45:45.255 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-08T16:45:45.256 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-05-08T16:45:45.256 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-05-08T16:45:45.257 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-05-08T16:45:45.258 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-05-08T16:45:45.258 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-05-08T16:45:45.259 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-05-08T16:45:45.260 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-08T16:45:45.260 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-05-08T16:45:45.260 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-05-08T16:45:45.260 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-08T16:45:45.261 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-05-08T16:45:45.262 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-05-08T16:45:45.262 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-eio 2024-05-08T16:45:45.264 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-05-08T16:45:45.264 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T16:45:45.264 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81110 2024-05-08T16:45:45.264 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.81110 2024-05-08T16:45:45.265 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-05-08T16:45:45.265 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-05-08T16:45:45.265 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/test-erasure-eio 2024-05-08T16:45:45.267 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-05-08T16:45:45.267 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T16:45:45.267 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81110 2024-05-08T16:45:45.267 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.81110 2024-05-08T16:45:45.269 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-05-08T16:45:45.269 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-05-08T16:45:45.269 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-05-08T16:45:45.269 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/test-erasure-eio 1' TERM HUP INT 2024-05-08T16:45:45.269 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:34: run: run_mon td/test-erasure-eio a 2024-05-08T16:45:45.269 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/test-erasure-eio 2024-05-08T16:45:45.269 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-05-08T16:45:45.269 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-05-08T16:45:45.269 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-05-08T16:45:45.269 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/test-erasure-eio/a 2024-05-08T16:45:45.269 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-eio/a --run-dir=td/test-erasure-eio 2024-05-08T16:45:45.680 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-05-08T16:45:45.680 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-08T16:45:45.680 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-08T16:45:45.681 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-08T16:45:45.681 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T16:45:45.681 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81110 2024-05-08T16:45:45.682 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.81110/$cluster-$name.asok' 2024-05-08T16:45:45.682 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-eio/a '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.81110/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-eio/log --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-05-08T16:45:45.906 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-05-08T16:45:45.907 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-05-08T16:45:45.907 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-05-08T16:45:45.907 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-05-08T16:45:45.907 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-05-08T16:45:45.908 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-05-08T16:45:45.908 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-05-08T16:45:45.908 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-05-08T16:45:45.908 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-05-08T16:45:45.909 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-08T16:45:45.909 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T16:45:45.909 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81110 2024-05-08T16:45:45.909 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.81110/ceph-mon.a.asok 2024-05-08T16:45:45.909 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-05-08T16:45:45.910 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.81110/ceph-mon.a.asok config get fsid 2024-05-08T16:45:46.028 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-05-08T16:45:46.028 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-05-08T16:45:46.028 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-05-08T16:45:46.028 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-05-08T16:45:46.028 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-05-08T16:45:46.028 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-05-08T16:45:46.028 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-05-08T16:45:46.028 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-05-08T16:45:46.028 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-08T16:45:46.028 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T16:45:46.029 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81110 2024-05-08T16:45:46.029 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.81110/ceph-mon.a.asok 2024-05-08T16:45:46.029 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-05-08T16:45:46.030 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.81110/ceph-mon.a.asok config get mon_host 2024-05-08T16:45:46.133 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:35: run: run_mgr td/test-erasure-eio x 2024-05-08T16:45:46.134 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/test-erasure-eio 2024-05-08T16:45:46.134 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-05-08T16:45:46.134 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-05-08T16:45:46.134 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-05-08T16:45:46.134 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/test-erasure-eio/x 2024-05-08T16:45:46.134 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-05-08T16:45:47.073 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-05-08T16:45:47.074 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-08T16:45:47.074 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-08T16:45:47.074 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-08T16:45:47.074 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T16:45:47.074 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81110 2024-05-08T16:45:47.074 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.81110/$cluster-$name.asok' 2024-05-08T16:45:47.075 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-05-08T16:45:47.077 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-eio/x '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.81110/$cluster-$name.asok' --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-05-08T16:45:47.111 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:36: run: create_pool rbd 4 2024-05-08T16:45:47.112 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-05-08T16:45:47.444 INFO:tasks.workunit.client.0.smithi172.stderr:pool 'rbd' already exists 2024-05-08T16:45:47.461 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-05-08T16:45:48.464 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: get_asok_path mon.a 2024-05-08T16:45:48.465 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-05-08T16:45:48.465 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-05-08T16:45:48.465 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-08T16:45:48.465 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T16:45:48.465 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81110 2024-05-08T16:45:48.465 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.81110/ceph-mon.a.asok 2024-05-08T16:45:48.465 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: CEPH_ARGS= 2024-05-08T16:45:48.466 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: ceph --admin-daemon /tmp/ceph-asok.81110/ceph-mon.a.asok log flush 2024-05-08T16:45:48.580 INFO:tasks.workunit.client.0.smithi172.stdout:{} 2024-05-08T16:45:48.593 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:40: run: grep 'load: jerasure.*lrc' td/test-erasure-eio/mon.a.log 2024-05-08T16:45:48.596 INFO:tasks.workunit.client.0.smithi172.stdout:2024-05-08T16:45:45.728+0000 7fe76ff18c80 0 load: jerasure load: lrc 2024-05-08T16:45:48.597 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:41: run: TEST_ec_recovery_unfound td/test-erasure-eio 2024-05-08T16:45:48.597 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:609: TEST_ec_recovery_unfound: local dir=td/test-erasure-eio 2024-05-08T16:45:48.597 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:610: TEST_ec_recovery_unfound: local objname=myobject 2024-05-08T16:45:48.597 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:611: TEST_ec_recovery_unfound: local lastobj=100 2024-05-08T16:45:48.597 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:613: TEST_ec_recovery_unfound: local testobj=obj75 2024-05-08T16:45:48.597 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:615: TEST_ec_recovery_unfound: ORIG_ARGS='--fsid=4457b0f1-028b-482b-8ee3-7ee05861e6cc --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-05-08T16:45:48.597 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:616: TEST_ec_recovery_unfound: CEPH_ARGS+=' --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 ' 2024-05-08T16:45:48.597 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:617: TEST_ec_recovery_unfound: CEPH_ARGS+=' --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10' 2024-05-08T16:45:48.597 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:618: TEST_ec_recovery_unfound: setup_osds 5 2024-05-08T16:45:48.597 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:47: setup_osds: local count=5 2024-05-08T16:45:48.597 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:48: setup_osds: shift 2024-05-08T16:45:48.598 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: expr 5 - 1 2024-05-08T16:45:48.599 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: seq 0 4 2024-05-08T16:45:48.601 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-05-08T16:45:48.601 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 0 2024-05-08T16:45:48.602 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-05-08T16:45:48.602 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-05-08T16:45:48.602 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-05-08T16:45:48.602 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-05-08T16:45:48.602 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/0 2024-05-08T16:45:48.602 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=4457b0f1-028b-482b-8ee3-7ee05861e6cc --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10' 2024-05-08T16:45:48.602 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-08T16:45:48.602 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-05-08T16:45:48.602 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-08T16:45:48.602 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/0' 2024-05-08T16:45:48.602 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/0/journal' 2024-05-08T16:45:48.602 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-05-08T16:45:48.602 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-05-08T16:45:48.603 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-05-08T16:45:48.604 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-05-08T16:45:48.604 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-08T16:45:48.605 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-08T16:45:48.605 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-08T16:45:48.605 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T16:45:48.605 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81110 2024-05-08T16:45:48.605 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.81110/$cluster-$name.asok' 2024-05-08T16:45:48.605 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.81110/$cluster-$name.asok' 2024-05-08T16:45:48.605 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-05-08T16:45:48.605 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-05-08T16:45:48.605 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-05-08T16:45:48.605 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-05-08T16:45:48.605 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-05-08T16:45:48.605 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-08T16:45:48.605 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-08T16:45:48.605 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-08T16:45:48.606 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-08T16:45:48.606 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-05-08T16:45:48.606 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-05-08T16:45:48.606 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/0 2024-05-08T16:45:48.606 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-05-08T16:45:48.607 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=596a7472-e93b-4226-891b-c5e25b76dd58 2024-05-08T16:45:48.607 INFO:tasks.workunit.client.0.smithi172.stdout:add osd0 596a7472-e93b-4226-891b-c5e25b76dd58 2024-05-08T16:45:48.607 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 596a7472-e93b-4226-891b-c5e25b76dd58' 2024-05-08T16:45:48.608 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-05-08T16:45:48.631 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQA8rDtmsFr5JRAAOywGcR3pKDlO2MhV3Ksg6w== 2024-05-08T16:45:48.631 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQA8rDtmsFr5JRAAOywGcR3pKDlO2MhV3Ksg6w=="}' 2024-05-08T16:45:48.631 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 596a7472-e93b-4226-891b-c5e25b76dd58 -i td/test-erasure-eio/0/new.json 2024-05-08T16:45:48.962 INFO:tasks.workunit.client.0.smithi172.stdout:0 2024-05-08T16:45:48.978 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/0/new.json 2024-05-08T16:45:48.979 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=4457b0f1-028b-482b-8ee3-7ee05861e6cc --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81110/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQA8rDtmsFr5JRAAOywGcR3pKDlO2MhV3Ksg6w== --osd-uuid 596a7472-e93b-4226-891b-c5e25b76dd58 2024-05-08T16:45:49.027 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:45:49.028+0000 7f0d06a217c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:45:49.039 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:45:49.040+0000 7f0d06a217c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:45:49.040 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:45:49.040+0000 7f0d06a217c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:45:49.041 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:45:49.044+0000 7f0d06a217c0 -1 bluestore(td/test-erasure-eio/0/block) _read_bdev_label failed to open td/test-erasure-eio/0/block: (2) No such file or directory 2024-05-08T16:45:49.041 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:45:49.044+0000 7f0d06a217c0 -1 bluestore(td/test-erasure-eio/0/block) _read_bdev_label failed to open td/test-erasure-eio/0/block: (2) No such file or directory 2024-05-08T16:45:49.041 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:45:49.044+0000 7f0d06a217c0 -1 bluestore(td/test-erasure-eio/0/block) _read_bdev_label failed to open td/test-erasure-eio/0/block: (2) No such file or directory 2024-05-08T16:45:49.079 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:45:49.080+0000 7f0d06a217c0 -1 bluestore(td/test-erasure-eio/0) _read_fsid unparsable uuid 2024-05-08T16:45:56.857 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/0/keyring 2024-05-08T16:45:56.857 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-05-08T16:45:56.858 INFO:tasks.workunit.client.0.smithi172.stdout:adding osd0 key to auth repository 2024-05-08T16:45:56.859 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-05-08T16:45:56.859 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-05-08T16:45:57.431 INFO:tasks.workunit.client.0.smithi172.stdout:start osd.0 2024-05-08T16:45:57.432 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-05-08T16:45:57.432 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=4457b0f1-028b-482b-8ee3-7ee05861e6cc --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81110/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-05-08T16:45:57.432 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-05-08T16:45:57.432 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-05-08T16:45:57.432 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-05-08T16:45:57.465 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:45:57.468+0000 7f28c38457c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:45:57.467 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:45:57.468+0000 7f28c38457c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:45:57.468 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:45:57.468+0000 7f28c38457c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:45:57.833 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-05-08T16:45:57.834 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-08T16:45:57.834 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-05-08T16:45:57.834 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-08T16:45:57.834 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-08T16:45:57.834 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:45:57.834 INFO:tasks.workunit.client.0.smithi172.stdout:0 2024-05-08T16:45:57.835 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-08T16:45:57.835 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:45:57.835 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:45:58.042 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:45:58.044+0000 7f28c38457c0 -1 Falling back to public interface 2024-05-08T16:45:58.255 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:45:59.257 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:45:59.257 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:45:59.257 INFO:tasks.workunit.client.0.smithi172.stdout:1 2024-05-08T16:45:59.258 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-08T16:45:59.258 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:45:59.258 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:45:59.688 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:46:00.569 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:46:00.572+0000 7f28c38457c0 -1 osd.0 0 log_to_monitors true 2024-05-08T16:46:00.690 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:46:00.690 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:46:00.690 INFO:tasks.workunit.client.0.smithi172.stdout:2 2024-05-08T16:46:00.691 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-08T16:46:00.691 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:46:00.691 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:46:01.414 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:46:02.416 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:46:02.416 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:46:02.417 INFO:tasks.workunit.client.0.smithi172.stdout:3 2024-05-08T16:46:02.418 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-05-08T16:46:02.418 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:46:02.418 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:46:02.884 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:46:03.885 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:46:03.885 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:46:03.886 INFO:tasks.workunit.client.0.smithi172.stdout:4 2024-05-08T16:46:03.886 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-05-08T16:46:03.886 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:46:03.886 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:46:04.322 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:46:05.324 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:46:05.324 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:46:05.324 INFO:tasks.workunit.client.0.smithi172.stdout:5 2024-05-08T16:46:05.325 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-05-08T16:46:05.325 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:46:05.325 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:46:05.870 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:46:06.872 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:46:06.872 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:46:06.872 INFO:tasks.workunit.client.0.smithi172.stdout:6 2024-05-08T16:46:06.873 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-05-08T16:46:06.873 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:46:06.873 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:46:07.581 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:46:08.582 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:46:08.583 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:46:08.583 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-05-08T16:46:08.583 INFO:tasks.workunit.client.0.smithi172.stdout:7 2024-05-08T16:46:08.583 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:46:08.583 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:46:09.028 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:46:10.030 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:46:10.030 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:46:10.030 INFO:tasks.workunit.client.0.smithi172.stdout:8 2024-05-08T16:46:10.031 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-05-08T16:46:10.031 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:46:10.031 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:46:10.593 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:46:11.594 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:46:11.594 INFO:tasks.workunit.client.0.smithi172.stdout:9 2024-05-08T16:46:11.595 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:46:11.595 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-05-08T16:46:11.595 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:46:11.595 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:46:12.211 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:46:13.213 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:46:13.213 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:46:13.213 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-05-08T16:46:13.213 INFO:tasks.workunit.client.0.smithi172.stdout:10 2024-05-08T16:46:13.214 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:46:13.214 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:46:13.716 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:46:14.718 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:46:14.718 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:46:14.719 INFO:tasks.workunit.client.0.smithi172.stdout:11 2024-05-08T16:46:14.719 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-05-08T16:46:14.719 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:46:14.719 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:46:15.200 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:46:16.202 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:46:16.202 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:46:16.203 INFO:tasks.workunit.client.0.smithi172.stdout:12 2024-05-08T16:46:16.203 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-05-08T16:46:16.203 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:46:16.203 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:46:17.303 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:46:18.305 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:46:18.305 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:46:18.306 INFO:tasks.workunit.client.0.smithi172.stdout:13 2024-05-08T16:46:18.306 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-05-08T16:46:18.306 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:46:18.306 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:46:19.211 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:46:20.213 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:46:20.213 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:46:20.213 INFO:tasks.workunit.client.0.smithi172.stdout:14 2024-05-08T16:46:20.214 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-05-08T16:46:20.214 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:46:20.214 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:46:21.218 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:46:22.219 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:46:22.219 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:46:22.220 INFO:tasks.workunit.client.0.smithi172.stdout:15 2024-05-08T16:46:22.220 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-05-08T16:46:22.220 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:46:22.220 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:46:23.395 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:46:24.397 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:46:24.397 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:46:24.397 INFO:tasks.workunit.client.0.smithi172.stdout:16 2024-05-08T16:46:24.398 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-05-08T16:46:24.398 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:46:24.398 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:46:25.434 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:46:26.436 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:46:26.436 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:46:26.437 INFO:tasks.workunit.client.0.smithi172.stdout:17 2024-05-08T16:46:26.437 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-05-08T16:46:26.437 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:46:26.437 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:46:27.755 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:46:28.757 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:46:28.757 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:46:28.757 INFO:tasks.workunit.client.0.smithi172.stdout:18 2024-05-08T16:46:28.758 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-05-08T16:46:28.758 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:46:28.758 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:46:29.290 INFO:tasks.workunit.client.0.smithi172.stdout:osd.0 up in weight 1 up_from 6 up_thru 6 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1149707749,v1:127.0.0.1:6803/1149707749] [v2:127.0.0.1:6804/1149707749,v1:127.0.0.1:6805/1149707749] exists,up 596a7472-e93b-4226-891b-c5e25b76dd58 2024-05-08T16:46:29.294 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-08T16:46:29.294 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-08T16:46:29.294 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-08T16:46:29.294 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-05-08T16:46:29.294 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 1 2024-05-08T16:46:29.294 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-05-08T16:46:29.294 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-05-08T16:46:29.294 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-05-08T16:46:29.294 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-05-08T16:46:29.294 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/1 2024-05-08T16:46:29.294 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=4457b0f1-028b-482b-8ee3-7ee05861e6cc --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10' 2024-05-08T16:46:29.294 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-08T16:46:29.294 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-05-08T16:46:29.295 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-08T16:46:29.295 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/1' 2024-05-08T16:46:29.295 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/1/journal' 2024-05-08T16:46:29.295 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-05-08T16:46:29.295 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-05-08T16:46:29.295 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-05-08T16:46:29.295 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-05-08T16:46:29.295 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-08T16:46:29.295 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-08T16:46:29.295 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-08T16:46:29.295 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T16:46:29.295 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81110 2024-05-08T16:46:29.295 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.81110/$cluster-$name.asok' 2024-05-08T16:46:29.295 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.81110/$cluster-$name.asok' 2024-05-08T16:46:29.296 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-05-08T16:46:29.296 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-05-08T16:46:29.296 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-05-08T16:46:29.296 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-05-08T16:46:29.296 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-05-08T16:46:29.296 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-08T16:46:29.296 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-08T16:46:29.296 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-08T16:46:29.296 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-08T16:46:29.296 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-05-08T16:46:29.296 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-05-08T16:46:29.296 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/1 2024-05-08T16:46:29.297 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-05-08T16:46:29.297 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=e2b88b07-cce8-49c5-aebd-522b052332c4 2024-05-08T16:46:29.297 INFO:tasks.workunit.client.0.smithi172.stdout:add osd1 e2b88b07-cce8-49c5-aebd-522b052332c4 2024-05-08T16:46:29.297 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 e2b88b07-cce8-49c5-aebd-522b052332c4' 2024-05-08T16:46:29.298 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-05-08T16:46:29.329 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBlrDtmVsfoExAAuD+X9Lfi3zLd7rVxt6xGOg== 2024-05-08T16:46:29.329 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBlrDtmVsfoExAAuD+X9Lfi3zLd7rVxt6xGOg=="}' 2024-05-08T16:46:29.329 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new e2b88b07-cce8-49c5-aebd-522b052332c4 -i td/test-erasure-eio/1/new.json 2024-05-08T16:46:30.107 INFO:tasks.workunit.client.0.smithi172.stdout:1 2024-05-08T16:46:30.124 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/1/new.json 2024-05-08T16:46:30.125 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=4457b0f1-028b-482b-8ee3-7ee05861e6cc --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81110/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBlrDtmVsfoExAAuD+X9Lfi3zLd7rVxt6xGOg== --osd-uuid e2b88b07-cce8-49c5-aebd-522b052332c4 2024-05-08T16:46:30.160 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:46:30.159+0000 7f00fc4e07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:46:30.197 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:46:30.199+0000 7f00fc4e07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:46:30.199 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:46:30.199+0000 7f00fc4e07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:46:30.200 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:46:30.199+0000 7f00fc4e07c0 -1 bluestore(td/test-erasure-eio/1/block) _read_bdev_label failed to open td/test-erasure-eio/1/block: (2) No such file or directory 2024-05-08T16:46:30.200 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:46:30.199+0000 7f00fc4e07c0 -1 bluestore(td/test-erasure-eio/1/block) _read_bdev_label failed to open td/test-erasure-eio/1/block: (2) No such file or directory 2024-05-08T16:46:30.200 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:46:30.199+0000 7f00fc4e07c0 -1 bluestore(td/test-erasure-eio/1/block) _read_bdev_label failed to open td/test-erasure-eio/1/block: (2) No such file or directory 2024-05-08T16:46:30.237 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:46:30.239+0000 7f00fc4e07c0 -1 bluestore(td/test-erasure-eio/1) _read_fsid unparsable uuid 2024-05-08T16:46:37.629 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/1/keyring 2024-05-08T16:46:37.630 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-05-08T16:46:37.631 INFO:tasks.workunit.client.0.smithi172.stdout:adding osd1 key to auth repository 2024-05-08T16:46:37.632 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-05-08T16:46:37.632 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-05-08T16:46:38.229 INFO:tasks.workunit.client.0.smithi172.stdout:start osd.1 2024-05-08T16:46:38.230 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-05-08T16:46:38.230 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=4457b0f1-028b-482b-8ee3-7ee05861e6cc --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81110/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-05-08T16:46:38.230 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-05-08T16:46:38.230 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-05-08T16:46:38.231 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-05-08T16:46:38.269 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:46:38.275+0000 7f01ee9187c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:46:38.310 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:46:38.315+0000 7f01ee9187c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:46:38.312 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:46:38.315+0000 7f01ee9187c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:46:38.676 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-05-08T16:46:38.676 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-08T16:46:38.676 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-05-08T16:46:38.676 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-08T16:46:38.676 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-08T16:46:38.676 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:46:38.676 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-08T16:46:38.677 INFO:tasks.workunit.client.0.smithi172.stdout:0 2024-05-08T16:46:38.677 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:46:38.677 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T16:46:38.898 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:46:38.903+0000 7f01ee9187c0 -1 Falling back to public interface 2024-05-08T16:46:39.120 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:46:40.122 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:46:40.122 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:46:40.122 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-08T16:46:40.123 INFO:tasks.workunit.client.0.smithi172.stdout:1 2024-05-08T16:46:40.123 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:46:40.123 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T16:46:40.616 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:46:41.617 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:46:41.617 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:46:41.617 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-08T16:46:41.617 INFO:tasks.workunit.client.0.smithi172.stdout:2 2024-05-08T16:46:41.618 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:46:41.618 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T16:46:41.618 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:46:41.623+0000 7f01ee9187c0 -1 osd.1 0 log_to_monitors true 2024-05-08T16:46:42.043 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:46:43.045 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:46:43.045 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:46:43.045 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-05-08T16:46:43.046 INFO:tasks.workunit.client.0.smithi172.stdout:3 2024-05-08T16:46:43.046 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:46:43.046 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T16:46:43.522 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:46:44.523 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:46:44.523 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:46:44.523 INFO:tasks.workunit.client.0.smithi172.stdout:4 2024-05-08T16:46:44.524 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-05-08T16:46:44.524 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:46:44.524 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T16:46:45.086 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:46:46.088 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:46:46.088 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:46:46.088 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-05-08T16:46:46.088 INFO:tasks.workunit.client.0.smithi172.stdout:5 2024-05-08T16:46:46.089 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:46:46.089 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T16:46:46.660 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:46:47.662 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:46:47.662 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:46:47.663 INFO:tasks.workunit.client.0.smithi172.stdout:6 2024-05-08T16:46:47.663 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-05-08T16:46:47.663 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:46:47.663 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T16:46:48.190 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:46:49.192 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:46:49.192 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:46:49.193 INFO:tasks.workunit.client.0.smithi172.stdout:7 2024-05-08T16:46:49.193 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-05-08T16:46:49.193 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:46:49.193 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T16:46:49.814 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:46:50.816 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:46:50.817 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:46:50.817 INFO:tasks.workunit.client.0.smithi172.stdout:8 2024-05-08T16:46:50.818 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-05-08T16:46:50.818 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:46:50.818 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T16:46:51.393 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:46:52.395 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:46:52.395 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:46:52.395 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-05-08T16:46:52.395 INFO:tasks.workunit.client.0.smithi172.stdout:9 2024-05-08T16:46:52.396 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:46:52.396 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T16:46:52.922 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:46:53.924 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:46:53.924 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:46:53.924 INFO:tasks.workunit.client.0.smithi172.stdout:10 2024-05-08T16:46:53.925 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-05-08T16:46:53.925 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:46:53.925 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T16:46:54.516 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:46:55.518 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:46:55.518 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:46:55.518 INFO:tasks.workunit.client.0.smithi172.stdout:11 2024-05-08T16:46:55.519 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-05-08T16:46:55.519 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:46:55.519 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T16:46:56.039 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:46:57.042 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:46:57.042 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:46:57.043 INFO:tasks.workunit.client.0.smithi172.stdout:12 2024-05-08T16:46:57.043 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-05-08T16:46:57.043 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:46:57.043 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T16:46:57.495 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:46:58.497 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:46:58.497 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:46:58.497 INFO:tasks.workunit.client.0.smithi172.stdout:13 2024-05-08T16:46:58.498 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-05-08T16:46:58.498 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:46:58.498 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T16:46:59.356 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:47:00.357 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:47:00.357 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:47:00.358 INFO:tasks.workunit.client.0.smithi172.stdout:14 2024-05-08T16:47:00.358 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-05-08T16:47:00.358 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:47:00.358 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T16:47:01.333 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:47:02.335 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:47:02.335 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:47:02.335 INFO:tasks.workunit.client.0.smithi172.stdout:15 2024-05-08T16:47:02.336 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-05-08T16:47:02.336 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:47:02.336 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T16:47:03.221 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:47:04.222 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:47:04.223 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:47:04.223 INFO:tasks.workunit.client.0.smithi172.stdout:16 2024-05-08T16:47:04.223 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-05-08T16:47:04.224 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:47:04.224 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T16:47:05.285 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:47:06.287 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:47:06.287 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:47:06.287 INFO:tasks.workunit.client.0.smithi172.stdout:17 2024-05-08T16:47:06.288 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-05-08T16:47:06.288 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:47:06.288 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T16:47:07.237 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:47:08.239 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:47:08.239 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:47:08.239 INFO:tasks.workunit.client.0.smithi172.stdout:18 2024-05-08T16:47:08.240 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-05-08T16:47:08.240 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:47:08.240 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T16:47:09.031 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:47:10.032 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:47:10.033 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:47:10.033 INFO:tasks.workunit.client.0.smithi172.stdout:19 2024-05-08T16:47:10.033 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-05-08T16:47:10.033 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:47:10.033 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T16:47:10.947 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:47:11.949 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:47:11.949 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:47:11.949 INFO:tasks.workunit.client.0.smithi172.stdout:20 2024-05-08T16:47:11.950 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-05-08T16:47:11.950 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:47:11.950 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T16:47:12.504 INFO:tasks.workunit.client.0.smithi172.stdout:osd.1 up in weight 1 up_from 13 up_thru 13 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/2238140369,v1:127.0.0.1:6811/2238140369] [v2:127.0.0.1:6812/2238140369,v1:127.0.0.1:6813/2238140369] exists,up e2b88b07-cce8-49c5-aebd-522b052332c4 2024-05-08T16:47:12.507 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-08T16:47:12.507 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-08T16:47:12.508 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-08T16:47:12.508 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-05-08T16:47:12.508 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 2 2024-05-08T16:47:12.508 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-05-08T16:47:12.508 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-05-08T16:47:12.508 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-05-08T16:47:12.508 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-05-08T16:47:12.508 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/2 2024-05-08T16:47:12.508 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=4457b0f1-028b-482b-8ee3-7ee05861e6cc --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10' 2024-05-08T16:47:12.508 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-08T16:47:12.508 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-05-08T16:47:12.508 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-08T16:47:12.508 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/2' 2024-05-08T16:47:12.508 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/2/journal' 2024-05-08T16:47:12.509 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-05-08T16:47:12.509 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-05-08T16:47:12.509 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-05-08T16:47:12.509 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-05-08T16:47:12.509 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-08T16:47:12.509 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-08T16:47:12.509 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-08T16:47:12.509 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T16:47:12.509 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81110 2024-05-08T16:47:12.509 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.81110/$cluster-$name.asok' 2024-05-08T16:47:12.509 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.81110/$cluster-$name.asok' 2024-05-08T16:47:12.509 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-05-08T16:47:12.509 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-05-08T16:47:12.509 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-05-08T16:47:12.509 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-05-08T16:47:12.510 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-05-08T16:47:12.510 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-08T16:47:12.510 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-08T16:47:12.510 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-08T16:47:12.510 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-08T16:47:12.510 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-05-08T16:47:12.510 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-05-08T16:47:12.510 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/2 2024-05-08T16:47:12.511 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-05-08T16:47:12.511 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=da1c5da2-b226-476c-8ffd-001130f7d176 2024-05-08T16:47:12.511 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 da1c5da2-b226-476c-8ffd-001130f7d176' 2024-05-08T16:47:12.511 INFO:tasks.workunit.client.0.smithi172.stdout:add osd2 da1c5da2-b226-476c-8ffd-001130f7d176 2024-05-08T16:47:12.511 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-05-08T16:47:12.539 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCQrDtmY2R6IBAAbUrKzGTD3M1xjeXuWoY5UA== 2024-05-08T16:47:12.539 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCQrDtmY2R6IBAAbUrKzGTD3M1xjeXuWoY5UA=="}' 2024-05-08T16:47:12.539 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new da1c5da2-b226-476c-8ffd-001130f7d176 -i td/test-erasure-eio/2/new.json 2024-05-08T16:47:13.080 INFO:tasks.workunit.client.0.smithi172.stdout:2 2024-05-08T16:47:13.095 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/2/new.json 2024-05-08T16:47:13.097 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=4457b0f1-028b-482b-8ee3-7ee05861e6cc --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81110/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCQrDtmY2R6IBAAbUrKzGTD3M1xjeXuWoY5UA== --osd-uuid da1c5da2-b226-476c-8ffd-001130f7d176 2024-05-08T16:47:13.138 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:47:13.142+0000 7fcf40d4c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:47:13.140 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:47:13.142+0000 7fcf40d4c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:47:13.141 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:47:13.146+0000 7fcf40d4c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:47:13.142 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:47:13.146+0000 7fcf40d4c7c0 -1 bluestore(td/test-erasure-eio/2/block) _read_bdev_label failed to open td/test-erasure-eio/2/block: (2) No such file or directory 2024-05-08T16:47:13.142 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:47:13.146+0000 7fcf40d4c7c0 -1 bluestore(td/test-erasure-eio/2/block) _read_bdev_label failed to open td/test-erasure-eio/2/block: (2) No such file or directory 2024-05-08T16:47:13.142 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:47:13.146+0000 7fcf40d4c7c0 -1 bluestore(td/test-erasure-eio/2/block) _read_bdev_label failed to open td/test-erasure-eio/2/block: (2) No such file or directory 2024-05-08T16:47:13.286 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:47:13.290+0000 7fcf40d4c7c0 -1 bluestore(td/test-erasure-eio/2) _read_fsid unparsable uuid 2024-05-08T16:47:20.631 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/2/keyring 2024-05-08T16:47:20.631 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-05-08T16:47:20.632 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-05-08T16:47:20.633 INFO:tasks.workunit.client.0.smithi172.stdout:adding osd2 key to auth repository 2024-05-08T16:47:20.633 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-05-08T16:47:21.195 INFO:tasks.workunit.client.0.smithi172.stdout:start osd.2 2024-05-08T16:47:21.195 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-05-08T16:47:21.196 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=4457b0f1-028b-482b-8ee3-7ee05861e6cc --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81110/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-05-08T16:47:21.196 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-05-08T16:47:21.196 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-05-08T16:47:21.196 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-05-08T16:47:21.234 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:47:21.238+0000 7fed7161c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:47:21.277 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:47:21.282+0000 7fed7161c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:47:21.279 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:47:21.282+0000 7fed7161c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:47:21.670 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-05-08T16:47:21.670 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-08T16:47:21.670 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-05-08T16:47:21.670 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-08T16:47:21.670 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-08T16:47:21.670 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:47:21.671 INFO:tasks.workunit.client.0.smithi172.stdout:0 2024-05-08T16:47:21.671 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-08T16:47:21.671 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T16:47:21.671 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:47:21.850 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:47:21.854+0000 7fed7161c7c0 -1 Falling back to public interface 2024-05-08T16:47:22.156 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:47:23.158 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:47:23.158 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:47:23.158 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-08T16:47:23.158 INFO:tasks.workunit.client.0.smithi172.stdout:1 2024-05-08T16:47:23.159 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:47:23.159 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T16:47:23.570 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:47:24.571 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:47:24.571 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:47:24.572 INFO:tasks.workunit.client.0.smithi172.stdout:2 2024-05-08T16:47:24.572 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-08T16:47:24.572 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:47:24.572 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T16:47:24.653 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:47:24.654+0000 7fed7161c7c0 -1 osd.2 0 log_to_monitors true 2024-05-08T16:47:25.044 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:47:26.046 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:47:26.046 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:47:26.046 INFO:tasks.workunit.client.0.smithi172.stdout:3 2024-05-08T16:47:26.047 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-05-08T16:47:26.047 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:47:26.047 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T16:47:26.576 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:47:27.577 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:47:27.577 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:47:27.578 INFO:tasks.workunit.client.0.smithi172.stdout:4 2024-05-08T16:47:27.578 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-05-08T16:47:27.578 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:47:27.578 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T16:47:28.079 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:47:29.081 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:47:29.081 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:47:29.081 INFO:tasks.workunit.client.0.smithi172.stdout:5 2024-05-08T16:47:29.082 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-05-08T16:47:29.082 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:47:29.082 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T16:47:29.548 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:47:30.550 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:47:30.550 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:47:30.551 INFO:tasks.workunit.client.0.smithi172.stdout:6 2024-05-08T16:47:30.551 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-05-08T16:47:30.551 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:47:30.551 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T16:47:31.138 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:47:32.140 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:47:32.140 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:47:32.140 INFO:tasks.workunit.client.0.smithi172.stdout:7 2024-05-08T16:47:32.141 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-05-08T16:47:32.141 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:47:32.141 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T16:47:32.620 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:47:33.622 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:47:33.622 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:47:33.622 INFO:tasks.workunit.client.0.smithi172.stdout:8 2024-05-08T16:47:33.623 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-05-08T16:47:33.623 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:47:33.623 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T16:47:34.196 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:47:35.197 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:47:35.198 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:47:35.198 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-05-08T16:47:35.198 INFO:tasks.workunit.client.0.smithi172.stdout:9 2024-05-08T16:47:35.199 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:47:35.199 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T16:47:35.751 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:47:36.753 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:47:36.753 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:47:36.753 INFO:tasks.workunit.client.0.smithi172.stdout:10 2024-05-08T16:47:36.754 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-05-08T16:47:36.754 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:47:36.754 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T16:47:37.395 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:47:38.397 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:47:38.397 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:47:38.398 INFO:tasks.workunit.client.0.smithi172.stdout:11 2024-05-08T16:47:38.398 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-05-08T16:47:38.398 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:47:38.398 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T16:47:39.045 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:47:40.047 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:47:40.047 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:47:40.048 INFO:tasks.workunit.client.0.smithi172.stdout:12 2024-05-08T16:47:40.048 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-05-08T16:47:40.048 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:47:40.048 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T16:47:40.602 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:47:41.604 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:47:41.604 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:47:41.604 INFO:tasks.workunit.client.0.smithi172.stdout:13 2024-05-08T16:47:41.605 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-05-08T16:47:41.605 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:47:41.605 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T16:47:42.384 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:47:43.386 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:47:43.386 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:47:43.386 INFO:tasks.workunit.client.0.smithi172.stdout:14 2024-05-08T16:47:43.387 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-05-08T16:47:43.387 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:47:43.387 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T16:47:44.227 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:47:45.229 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:47:45.229 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:47:45.230 INFO:tasks.workunit.client.0.smithi172.stdout:15 2024-05-08T16:47:45.230 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-05-08T16:47:45.230 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:47:45.230 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T16:47:46.126 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:47:47.128 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:47:47.128 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:47:47.129 INFO:tasks.workunit.client.0.smithi172.stdout:16 2024-05-08T16:47:47.129 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-05-08T16:47:47.129 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:47:47.129 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T16:47:48.008 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:47:49.010 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:47:49.010 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:47:49.011 INFO:tasks.workunit.client.0.smithi172.stdout:17 2024-05-08T16:47:49.011 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-05-08T16:47:49.011 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:47:49.011 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T16:47:49.820 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:47:50.822 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:47:50.822 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:47:50.822 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-05-08T16:47:50.822 INFO:tasks.workunit.client.0.smithi172.stdout:18 2024-05-08T16:47:50.823 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:47:50.823 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T16:47:52.024 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:47:53.026 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:47:53.026 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:47:53.026 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-05-08T16:47:53.026 INFO:tasks.workunit.client.0.smithi172.stdout:19 2024-05-08T16:47:53.027 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:47:53.027 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T16:47:53.695 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:47:54.697 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:47:54.697 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:47:54.697 INFO:tasks.workunit.client.0.smithi172.stdout:20 2024-05-08T16:47:54.698 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-05-08T16:47:54.698 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:47:54.698 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T16:47:55.359 INFO:tasks.workunit.client.0.smithi172.stdout:osd.2 up in weight 1 up_from 19 up_thru 19 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/653306806,v1:127.0.0.1:6819/653306806] [v2:127.0.0.1:6820/653306806,v1:127.0.0.1:6821/653306806] exists,up da1c5da2-b226-476c-8ffd-001130f7d176 2024-05-08T16:47:55.361 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-08T16:47:55.362 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-08T16:47:55.362 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-08T16:47:55.362 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-05-08T16:47:55.362 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 3 2024-05-08T16:47:55.362 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-05-08T16:47:55.362 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-05-08T16:47:55.362 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2024-05-08T16:47:55.362 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-05-08T16:47:55.362 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/3 2024-05-08T16:47:55.362 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=4457b0f1-028b-482b-8ee3-7ee05861e6cc --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10' 2024-05-08T16:47:55.362 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-08T16:47:55.362 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-05-08T16:47:55.362 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-08T16:47:55.363 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/3' 2024-05-08T16:47:55.363 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/3/journal' 2024-05-08T16:47:55.363 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-05-08T16:47:55.363 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-05-08T16:47:55.363 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-05-08T16:47:55.363 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-05-08T16:47:55.363 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-08T16:47:55.363 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-08T16:47:55.363 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-08T16:47:55.363 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T16:47:55.363 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81110 2024-05-08T16:47:55.363 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.81110/$cluster-$name.asok' 2024-05-08T16:47:55.363 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.81110/$cluster-$name.asok' 2024-05-08T16:47:55.363 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-05-08T16:47:55.363 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-05-08T16:47:55.364 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-05-08T16:47:55.364 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-05-08T16:47:55.364 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-05-08T16:47:55.364 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-08T16:47:55.364 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-08T16:47:55.364 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-08T16:47:55.364 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-08T16:47:55.364 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-05-08T16:47:55.364 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-05-08T16:47:55.364 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/3 2024-05-08T16:47:55.364 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-05-08T16:47:55.365 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=498b0e02-d6e4-4f34-b403-e5672822cc63 2024-05-08T16:47:55.365 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 498b0e02-d6e4-4f34-b403-e5672822cc63' 2024-05-08T16:47:55.365 INFO:tasks.workunit.client.0.smithi172.stdout:add osd3 498b0e02-d6e4-4f34-b403-e5672822cc63 2024-05-08T16:47:55.366 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-05-08T16:47:55.396 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQC7rDtmHG7tFxAAQOC6G63ynKMLwCHiizn1oA== 2024-05-08T16:47:55.396 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQC7rDtmHG7tFxAAQOC6G63ynKMLwCHiizn1oA=="}' 2024-05-08T16:47:55.396 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 498b0e02-d6e4-4f34-b403-e5672822cc63 -i td/test-erasure-eio/3/new.json 2024-05-08T16:47:55.997 INFO:tasks.workunit.client.0.smithi172.stdout:3 2024-05-08T16:47:56.014 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/3/new.json 2024-05-08T16:47:56.016 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=4457b0f1-028b-482b-8ee3-7ee05861e6cc --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81110/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQC7rDtmHG7tFxAAQOC6G63ynKMLwCHiizn1oA== --osd-uuid 498b0e02-d6e4-4f34-b403-e5672822cc63 2024-05-08T16:47:56.058 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:47:56.061+0000 7f28beee87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:47:56.060 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:47:56.061+0000 7f28beee87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:47:56.062 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:47:56.065+0000 7f28beee87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:47:56.062 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:47:56.065+0000 7f28beee87c0 -1 bluestore(td/test-erasure-eio/3/block) _read_bdev_label failed to open td/test-erasure-eio/3/block: (2) No such file or directory 2024-05-08T16:47:56.062 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:47:56.065+0000 7f28beee87c0 -1 bluestore(td/test-erasure-eio/3/block) _read_bdev_label failed to open td/test-erasure-eio/3/block: (2) No such file or directory 2024-05-08T16:47:56.062 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:47:56.065+0000 7f28beee87c0 -1 bluestore(td/test-erasure-eio/3/block) _read_bdev_label failed to open td/test-erasure-eio/3/block: (2) No such file or directory 2024-05-08T16:47:56.173 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:47:56.177+0000 7f28beee87c0 -1 bluestore(td/test-erasure-eio/3) _read_fsid unparsable uuid 2024-05-08T16:48:03.474 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/3/keyring 2024-05-08T16:48:03.474 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-05-08T16:48:03.474 INFO:tasks.workunit.client.0.smithi172.stdout:adding osd3 key to auth repository 2024-05-08T16:48:03.474 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2024-05-08T16:48:03.475 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-05-08T16:48:04.057 INFO:tasks.workunit.client.0.smithi172.stdout:start osd.3 2024-05-08T16:48:04.058 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2024-05-08T16:48:04.058 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=4457b0f1-028b-482b-8ee3-7ee05861e6cc --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81110/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-05-08T16:48:04.058 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-05-08T16:48:04.058 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-05-08T16:48:04.058 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-05-08T16:48:04.097 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:48:04.097+0000 7fc5a9c267c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:48:04.139 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:48:04.141+0000 7fc5a9c267c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:48:04.140 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:48:04.141+0000 7fc5a9c267c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:48:04.517 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2024-05-08T16:48:04.517 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-08T16:48:04.517 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-05-08T16:48:04.517 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-08T16:48:04.517 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-08T16:48:04.517 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:48:04.517 INFO:tasks.workunit.client.0.smithi172.stdout:0 2024-05-08T16:48:04.518 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-08T16:48:04.518 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:48:04.518 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T16:48:04.698 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:48:04.701+0000 7fc5a9c267c0 -1 Falling back to public interface 2024-05-08T16:48:04.950 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:48:05.952 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:48:05.952 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:48:05.952 INFO:tasks.workunit.client.0.smithi172.stdout:1 2024-05-08T16:48:05.953 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-08T16:48:05.953 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:48:05.953 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T16:48:06.379 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:48:07.381 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:48:07.381 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:48:07.381 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-08T16:48:07.381 INFO:tasks.workunit.client.0.smithi172.stdout:2 2024-05-08T16:48:07.382 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:48:07.381+0000 7fc5a9c267c0 -1 osd.3 0 log_to_monitors true 2024-05-08T16:48:07.382 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:48:07.382 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T16:48:08.036 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:48:09.037 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:48:09.038 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:48:09.038 INFO:tasks.workunit.client.0.smithi172.stdout:3 2024-05-08T16:48:09.039 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-05-08T16:48:09.039 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:48:09.039 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T16:48:09.674 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:48:10.676 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:48:10.676 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:48:10.676 INFO:tasks.workunit.client.0.smithi172.stdout:4 2024-05-08T16:48:10.677 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-05-08T16:48:10.677 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:48:10.677 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T16:48:11.346 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:48:12.347 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:48:12.347 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:48:12.348 INFO:tasks.workunit.client.0.smithi172.stdout:5 2024-05-08T16:48:12.348 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-05-08T16:48:12.348 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:48:12.348 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T16:48:12.929 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:48:13.931 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:48:13.932 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:48:13.932 INFO:tasks.workunit.client.0.smithi172.stdout:6 2024-05-08T16:48:13.932 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-05-08T16:48:13.933 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:48:13.933 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T16:48:14.563 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:48:15.565 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:48:15.566 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:48:15.566 INFO:tasks.workunit.client.0.smithi172.stdout:7 2024-05-08T16:48:15.566 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-05-08T16:48:15.567 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:48:15.567 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T16:48:16.080 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:48:17.081 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:48:17.082 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:48:17.082 INFO:tasks.workunit.client.0.smithi172.stdout:8 2024-05-08T16:48:17.082 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-05-08T16:48:17.082 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:48:17.083 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T16:48:17.644 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:48:18.645 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:48:18.645 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:48:18.646 INFO:tasks.workunit.client.0.smithi172.stdout:9 2024-05-08T16:48:18.646 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-05-08T16:48:18.646 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:48:18.646 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T16:48:19.154 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:48:20.156 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:48:20.156 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:48:20.156 INFO:tasks.workunit.client.0.smithi172.stdout:10 2024-05-08T16:48:20.157 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-05-08T16:48:20.157 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:48:20.157 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T16:48:20.773 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:48:21.775 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:48:21.775 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:48:21.775 INFO:tasks.workunit.client.0.smithi172.stdout:11 2024-05-08T16:48:21.776 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-05-08T16:48:21.776 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:48:21.776 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T16:48:22.352 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:48:23.354 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:48:23.354 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:48:23.354 INFO:tasks.workunit.client.0.smithi172.stdout:12 2024-05-08T16:48:23.355 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-05-08T16:48:23.355 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:48:23.355 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T16:48:23.798 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:48:24.800 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:48:24.800 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:48:24.800 INFO:tasks.workunit.client.0.smithi172.stdout:13 2024-05-08T16:48:24.801 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-05-08T16:48:24.801 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:48:24.801 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T16:48:25.704 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:48:26.706 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:48:26.707 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:48:26.707 INFO:tasks.workunit.client.0.smithi172.stdout:14 2024-05-08T16:48:26.707 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-05-08T16:48:26.707 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:48:26.708 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T16:48:27.482 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:48:28.485 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:48:28.485 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:48:28.485 INFO:tasks.workunit.client.0.smithi172.stdout:15 2024-05-08T16:48:28.486 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-05-08T16:48:28.486 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:48:28.486 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T16:48:29.891 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:48:30.893 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:48:30.894 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:48:30.894 INFO:tasks.workunit.client.0.smithi172.stdout:16 2024-05-08T16:48:30.895 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-05-08T16:48:30.895 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:48:30.895 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T16:48:31.490 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:48:32.493 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:48:32.493 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:48:32.493 INFO:tasks.workunit.client.0.smithi172.stdout:17 2024-05-08T16:48:32.493 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-05-08T16:48:32.494 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:48:32.494 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T16:48:33.453 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:48:34.455 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:48:34.455 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:48:34.455 INFO:tasks.workunit.client.0.smithi172.stdout:18 2024-05-08T16:48:34.456 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-05-08T16:48:34.456 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:48:34.456 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T16:48:35.853 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:48:36.855 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:48:36.855 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:48:36.856 INFO:tasks.workunit.client.0.smithi172.stdout:19 2024-05-08T16:48:36.856 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-05-08T16:48:36.856 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:48:36.856 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T16:48:37.524 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:48:38.526 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:48:38.526 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:48:38.527 INFO:tasks.workunit.client.0.smithi172.stdout:20 2024-05-08T16:48:38.527 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-05-08T16:48:38.527 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:48:38.527 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T16:48:39.303 INFO:tasks.workunit.client.0.smithi172.stdout:osd.3 up in weight 1 up_from 27 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/4196917910,v1:127.0.0.1:6827/4196917910] [v2:127.0.0.1:6828/4196917910,v1:127.0.0.1:6829/4196917910] exists,up 498b0e02-d6e4-4f34-b403-e5672822cc63 2024-05-08T16:48:39.307 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-08T16:48:39.307 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-08T16:48:39.307 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-08T16:48:39.307 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-05-08T16:48:39.307 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 4 2024-05-08T16:48:39.307 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-05-08T16:48:39.307 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-05-08T16:48:39.307 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=4 2024-05-08T16:48:39.308 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-05-08T16:48:39.308 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/4 2024-05-08T16:48:39.308 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=4457b0f1-028b-482b-8ee3-7ee05861e6cc --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10' 2024-05-08T16:48:39.308 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-08T16:48:39.308 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-05-08T16:48:39.308 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-08T16:48:39.308 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/4' 2024-05-08T16:48:39.308 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/4/journal' 2024-05-08T16:48:39.308 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-05-08T16:48:39.308 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-05-08T16:48:39.308 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-05-08T16:48:39.308 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-05-08T16:48:39.308 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-08T16:48:39.308 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-08T16:48:39.308 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-08T16:48:39.309 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T16:48:39.309 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81110 2024-05-08T16:48:39.309 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.81110/$cluster-$name.asok' 2024-05-08T16:48:39.309 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.81110/$cluster-$name.asok' 2024-05-08T16:48:39.309 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-05-08T16:48:39.309 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-05-08T16:48:39.309 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-05-08T16:48:39.309 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-05-08T16:48:39.309 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-05-08T16:48:39.309 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-08T16:48:39.309 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-08T16:48:39.309 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-08T16:48:39.309 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-08T16:48:39.310 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-05-08T16:48:39.310 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-05-08T16:48:39.310 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/4 2024-05-08T16:48:39.310 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-05-08T16:48:39.310 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=d8fcadb2-a442-44f4-8736-ded970a8934a 2024-05-08T16:48:39.310 INFO:tasks.workunit.client.0.smithi172.stdout:add osd4 d8fcadb2-a442-44f4-8736-ded970a8934a 2024-05-08T16:48:39.311 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd4 d8fcadb2-a442-44f4-8736-ded970a8934a' 2024-05-08T16:48:39.311 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-05-08T16:48:39.339 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDnrDtmbzuEFBAAaT6uI4+6t+WRuHwWiYWp1A== 2024-05-08T16:48:39.339 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDnrDtmbzuEFBAAaT6uI4+6t+WRuHwWiYWp1A=="}' 2024-05-08T16:48:39.339 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new d8fcadb2-a442-44f4-8736-ded970a8934a -i td/test-erasure-eio/4/new.json 2024-05-08T16:48:40.070 INFO:tasks.workunit.client.0.smithi172.stdout:4 2024-05-08T16:48:40.090 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/4/new.json 2024-05-08T16:48:40.092 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 4 --fsid=4457b0f1-028b-482b-8ee3-7ee05861e6cc --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/4 --osd-journal=td/test-erasure-eio/4/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81110/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDnrDtmbzuEFBAAaT6uI4+6t+WRuHwWiYWp1A== --osd-uuid d8fcadb2-a442-44f4-8736-ded970a8934a 2024-05-08T16:48:40.135 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:48:40.136+0000 7fa15ea627c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:48:40.366 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:48:40.368+0000 7fa15ea627c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:48:40.368 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:48:40.368+0000 7fa15ea627c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:48:40.369 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:48:40.372+0000 7fa15ea627c0 -1 bluestore(td/test-erasure-eio/4/block) _read_bdev_label failed to open td/test-erasure-eio/4/block: (2) No such file or directory 2024-05-08T16:48:40.369 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:48:40.372+0000 7fa15ea627c0 -1 bluestore(td/test-erasure-eio/4/block) _read_bdev_label failed to open td/test-erasure-eio/4/block: (2) No such file or directory 2024-05-08T16:48:40.369 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:48:40.372+0000 7fa15ea627c0 -1 bluestore(td/test-erasure-eio/4/block) _read_bdev_label failed to open td/test-erasure-eio/4/block: (2) No such file or directory 2024-05-08T16:48:40.549 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:48:40.552+0000 7fa15ea627c0 -1 bluestore(td/test-erasure-eio/4) _read_fsid unparsable uuid 2024-05-08T16:48:47.867 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/4/keyring 2024-05-08T16:48:47.868 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-05-08T16:48:47.869 INFO:tasks.workunit.client.0.smithi172.stdout:adding osd4 key to auth repository 2024-05-08T16:48:47.869 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd4 key to auth repository 2024-05-08T16:48:47.869 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-05-08T16:48:48.435 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.4 2024-05-08T16:48:48.435 INFO:tasks.workunit.client.0.smithi172.stdout:start osd.4 2024-05-08T16:48:48.435 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-05-08T16:48:48.436 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 4 --fsid=4457b0f1-028b-482b-8ee3-7ee05861e6cc --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-recovery-max-single-start 3 --osd-recovery-max-active 3 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/4 --osd-journal=td/test-erasure-eio/4/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81110/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-05-08T16:48:48.436 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-05-08T16:48:48.436 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-05-08T16:48:48.476 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:48:48.475+0000 7f56e485a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:48:48.506 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:48:48.507+0000 7f56e485a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:48:48.507 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:48:48.507+0000 7f56e485a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:48:48.852 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 4 2024-05-08T16:48:48.852 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-08T16:48:48.852 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=4 2024-05-08T16:48:48.852 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-08T16:48:48.852 INFO:tasks.workunit.client.0.smithi172.stdout:0 2024-05-08T16:48:48.853 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-08T16:48:48.853 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:48:48.853 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-08T16:48:48.853 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:48:48.853 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-08T16:48:49.068 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:48:49.067+0000 7f56e485a7c0 -1 Falling back to public interface 2024-05-08T16:48:49.279 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:48:50.281 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:48:50.281 INFO:tasks.workunit.client.0.smithi172.stdout:1 2024-05-08T16:48:50.282 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:48:50.282 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-08T16:48:50.282 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:48:50.282 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-08T16:48:50.694 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:48:51.696 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:48:51.696 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:48:51.696 INFO:tasks.workunit.client.0.smithi172.stdout:2 2024-05-08T16:48:51.697 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-08T16:48:51.697 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:48:51.697 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-08T16:48:51.883 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:48:51.883+0000 7f56e485a7c0 -1 osd.4 0 log_to_monitors true 2024-05-08T16:48:52.170 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:48:53.172 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:48:53.172 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:48:53.172 INFO:tasks.workunit.client.0.smithi172.stdout:3 2024-05-08T16:48:53.173 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-05-08T16:48:53.173 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:48:53.173 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-08T16:48:53.987 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:48:54.988 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:48:54.989 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:48:54.989 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-05-08T16:48:54.989 INFO:tasks.workunit.client.0.smithi172.stdout:4 2024-05-08T16:48:54.989 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:48:54.989 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-08T16:48:55.506 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:48:56.508 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:48:56.508 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:48:56.509 INFO:tasks.workunit.client.0.smithi172.stdout:5 2024-05-08T16:48:56.509 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-05-08T16:48:56.509 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:48:56.509 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-08T16:48:57.213 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:48:58.214 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:48:58.215 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:48:58.215 INFO:tasks.workunit.client.0.smithi172.stdout:6 2024-05-08T16:48:58.215 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-05-08T16:48:58.215 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:48:58.216 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-08T16:48:58.818 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:48:59.820 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:48:59.820 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:48:59.821 INFO:tasks.workunit.client.0.smithi172.stdout:7 2024-05-08T16:48:59.821 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-05-08T16:48:59.821 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:48:59.821 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-08T16:49:00.629 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:49:01.631 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:49:01.632 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:49:01.632 INFO:tasks.workunit.client.0.smithi172.stdout:8 2024-05-08T16:49:01.633 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-05-08T16:49:01.633 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:49:01.633 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-08T16:49:02.355 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:49:03.357 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:49:03.357 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:49:03.358 INFO:tasks.workunit.client.0.smithi172.stdout:9 2024-05-08T16:49:03.358 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-05-08T16:49:03.358 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:49:03.358 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-08T16:49:03.833 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:49:04.834 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:49:04.834 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:49:04.835 INFO:tasks.workunit.client.0.smithi172.stdout:10 2024-05-08T16:49:04.835 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-05-08T16:49:04.835 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:49:04.835 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-08T16:49:05.515 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:49:06.517 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:49:06.517 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:49:06.517 INFO:tasks.workunit.client.0.smithi172.stdout:11 2024-05-08T16:49:06.518 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-05-08T16:49:06.518 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:49:06.518 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-08T16:49:07.251 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:49:08.253 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:49:08.253 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:49:08.253 INFO:tasks.workunit.client.0.smithi172.stdout:12 2024-05-08T16:49:08.254 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-05-08T16:49:08.254 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:49:08.254 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-08T16:49:08.834 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:49:09.836 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:49:09.836 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:49:09.836 INFO:tasks.workunit.client.0.smithi172.stdout:13 2024-05-08T16:49:09.837 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-05-08T16:49:09.837 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:49:09.837 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-08T16:49:10.823 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:49:11.825 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:49:11.825 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:49:11.825 INFO:tasks.workunit.client.0.smithi172.stdout:14 2024-05-08T16:49:11.826 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-05-08T16:49:11.826 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:49:11.826 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-08T16:49:12.839 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:49:13.840 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:49:13.840 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:49:13.841 INFO:tasks.workunit.client.0.smithi172.stdout:15 2024-05-08T16:49:13.841 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-05-08T16:49:13.841 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:49:13.841 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-08T16:49:14.732 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:49:15.733 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:49:15.733 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:49:15.734 INFO:tasks.workunit.client.0.smithi172.stdout:16 2024-05-08T16:49:15.734 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-05-08T16:49:15.734 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:49:15.734 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-08T16:49:17.037 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:49:18.040 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:49:18.040 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:49:18.040 INFO:tasks.workunit.client.0.smithi172.stdout:17 2024-05-08T16:49:18.041 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-05-08T16:49:18.041 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:49:18.041 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-08T16:49:18.870 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:49:19.871 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:49:19.871 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:49:19.872 INFO:tasks.workunit.client.0.smithi172.stdout:18 2024-05-08T16:49:19.872 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-05-08T16:49:19.872 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:49:19.872 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-08T16:49:20.811 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:49:21.813 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:49:21.813 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:49:21.813 INFO:tasks.workunit.client.0.smithi172.stdout:19 2024-05-08T16:49:21.814 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-05-08T16:49:21.814 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:49:21.814 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-08T16:49:22.464 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:49:23.466 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:49:23.466 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:49:23.466 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-05-08T16:49:23.466 INFO:tasks.workunit.client.0.smithi172.stdout:20 2024-05-08T16:49:23.466 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:49:23.467 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-08T16:49:23.955 INFO:tasks.workunit.client.0.smithi172.stdout:osd.4 up in weight 1 up_from 38 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/2469440607,v1:127.0.0.1:6835/2469440607] [v2:127.0.0.1:6836/2469440607,v1:127.0.0.1:6837/2469440607] exists,up d8fcadb2-a442-44f4-8736-ded970a8934a 2024-05-08T16:49:23.956 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-08T16:49:23.956 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-08T16:49:23.956 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-08T16:49:23.956 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: get_asok_path osd.0 2024-05-08T16:49:23.956 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-05-08T16:49:23.956 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-05-08T16:49:23.957 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-08T16:49:23.957 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T16:49:23.957 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81110 2024-05-08T16:49:23.957 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.81110/ceph-osd.0.asok 2024-05-08T16:49:23.957 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: CEPH_ARGS= 2024-05-08T16:49:23.958 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: ceph --admin-daemon /tmp/ceph-asok.81110/ceph-osd.0.asok log flush 2024-05-08T16:49:24.068 INFO:tasks.workunit.client.0.smithi172.stdout:{} 2024-05-08T16:49:24.081 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:56: setup_osds: grep 'load: jerasure.*lrc' td/test-erasure-eio/osd.0.log 2024-05-08T16:49:24.086 INFO:tasks.workunit.client.0.smithi172.stdout:2024-05-08T16:45:58.048+0000 7f28c38457c0 0 load: jerasure load: lrc 2024-05-08T16:49:24.087 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:619: TEST_ec_recovery_unfound: CEPH_ARGS='--fsid=4457b0f1-028b-482b-8ee3-7ee05861e6cc --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-05-08T16:49:24.087 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:621: TEST_ec_recovery_unfound: local poolname=pool-jerasure 2024-05-08T16:49:24.087 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:622: TEST_ec_recovery_unfound: create_erasure_coded_pool pool-jerasure 3 2 2024-05-08T16:49:24.087 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:67: create_erasure_coded_pool: local poolname=pool-jerasure 2024-05-08T16:49:24.087 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:68: create_erasure_coded_pool: shift 2024-05-08T16:49:24.087 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:69: create_erasure_coded_pool: local k=3 2024-05-08T16:49:24.087 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:70: create_erasure_coded_pool: shift 2024-05-08T16:49:24.087 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:71: create_erasure_coded_pool: local m=2 2024-05-08T16:49:24.087 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:72: create_erasure_coded_pool: shift 2024-05-08T16:49:24.087 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:74: create_erasure_coded_pool: ceph osd erasure-code-profile set myprofile plugin=jerasure k=3 m=2 crush-failure-domain=osd 2024-05-08T16:49:25.070 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:78: create_erasure_coded_pool: create_pool pool-jerasure 1 1 erasure myprofile 2024-05-08T16:49:25.070 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create pool-jerasure 1 1 erasure myprofile 2024-05-08T16:49:26.652 INFO:tasks.workunit.client.0.smithi172.stderr:pool 'pool-jerasure' already exists 2024-05-08T16:49:26.670 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-05-08T16:49:27.673 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:80: create_erasure_coded_pool: wait_for_clean 2024-05-08T16:49:27.673 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-05-08T16:49:27.673 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-05-08T16:49:27.673 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-05-08T16:49:27.673 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-05-08T16:49:27.674 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-05-08T16:49:27.674 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-05-08T16:49:27.675 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-05-08T16:49:27.675 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-05-08T16:49:27.675 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-05-08T16:49:27.801 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-05-08T16:49:27.801 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-05-08T16:49:27.801 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-05-08T16:49:27.802 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-05-08T16:49:27.802 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-05-08T16:49:27.802 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-05-08T16:49:28.659 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-05-08T16:49:28.659 INFO:tasks.workunit.client.0.smithi172.stderr:1 2024-05-08T16:49:28.659 INFO:tasks.workunit.client.0.smithi172.stderr:2 2024-05-08T16:49:28.659 INFO:tasks.workunit.client.0.smithi172.stderr:3 2024-05-08T16:49:28.659 INFO:tasks.workunit.client.0.smithi172.stderr:4' 2024-05-08T16:49:28.659 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-05-08T16:49:28.659 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T16:49:28.660 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-05-08T16:49:28.833 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=25769803818 2024-05-08T16:49:28.834 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 25769803818 2024-05-08T16:49:28.834 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803818' 2024-05-08T16:49:28.834 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T16:49:28.834 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-05-08T16:49:29.030 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=55834574882 2024-05-08T16:49:29.031 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 55834574882 2024-05-08T16:49:29.031 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803818 1-55834574882' 2024-05-08T16:49:29.031 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T16:49:29.031 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-05-08T16:49:29.187 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=81604378652 2024-05-08T16:49:29.187 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 81604378652 2024-05-08T16:49:29.187 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803818 1-55834574882 2-81604378652' 2024-05-08T16:49:29.187 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T16:49:29.188 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-05-08T16:49:29.341 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=115964117009 2024-05-08T16:49:29.341 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 115964117009 2024-05-08T16:49:29.341 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803818 1-55834574882 2-81604378652 3-115964117009' 2024-05-08T16:49:29.341 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T16:49:29.341 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-05-08T16:49:29.483 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=163208757255 2024-05-08T16:49:29.483 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 163208757255 2024-05-08T16:49:29.484 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803818 1-55834574882 2-81604378652 3-115964117009 4-163208757255' 2024-05-08T16:49:29.484 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-08T16:49:29.484 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-25769803818 2024-05-08T16:49:29.484 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-08T16:49:29.484 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-05-08T16:49:29.485 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-25769803818 2024-05-08T16:49:29.485 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-08T16:49:29.486 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=25769803818 2024-05-08T16:49:29.486 INFO:tasks.workunit.client.0.smithi172.stdout:waiting osd.0 seq 25769803818 2024-05-08T16:49:29.486 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 25769803818' 2024-05-08T16:49:29.487 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-05-08T16:49:29.903 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 25769803819 -lt 25769803818 2024-05-08T16:49:29.903 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-08T16:49:29.904 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-55834574882 2024-05-08T16:49:29.905 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-08T16:49:29.907 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-05-08T16:49:29.907 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-55834574882 2024-05-08T16:49:29.907 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-08T16:49:29.908 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=55834574882 2024-05-08T16:49:29.908 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 55834574882' 2024-05-08T16:49:29.908 INFO:tasks.workunit.client.0.smithi172.stdout:waiting osd.1 seq 55834574882 2024-05-08T16:49:29.909 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-05-08T16:49:30.347 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 55834574882 -lt 55834574882 2024-05-08T16:49:30.347 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-08T16:49:30.348 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-81604378652 2024-05-08T16:49:30.348 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-08T16:49:30.349 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-05-08T16:49:30.350 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-81604378652 2024-05-08T16:49:30.350 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-08T16:49:30.351 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=81604378652 2024-05-08T16:49:30.351 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 81604378652' 2024-05-08T16:49:30.351 INFO:tasks.workunit.client.0.smithi172.stdout:waiting osd.2 seq 81604378652 2024-05-08T16:49:30.352 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-05-08T16:49:30.748 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 81604378653 -lt 81604378652 2024-05-08T16:49:30.748 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-08T16:49:30.749 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-115964117009 2024-05-08T16:49:30.749 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-08T16:49:30.750 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-05-08T16:49:30.751 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-08T16:49:30.751 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-115964117009 2024-05-08T16:49:30.752 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=115964117009 2024-05-08T16:49:30.752 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 115964117009' 2024-05-08T16:49:30.752 INFO:tasks.workunit.client.0.smithi172.stdout:waiting osd.3 seq 115964117009 2024-05-08T16:49:30.752 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-05-08T16:49:31.191 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 115964117009 -lt 115964117009 2024-05-08T16:49:31.191 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-08T16:49:31.192 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-163208757255 2024-05-08T16:49:31.192 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-08T16:49:31.194 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-05-08T16:49:31.195 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-163208757255 2024-05-08T16:49:31.195 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-08T16:49:31.196 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=163208757255 2024-05-08T16:49:31.197 INFO:tasks.workunit.client.0.smithi172.stdout:waiting osd.4 seq 163208757255 2024-05-08T16:49:31.197 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 163208757255' 2024-05-08T16:49:31.197 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-05-08T16:49:31.606 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 163208757255 -lt 163208757255 2024-05-08T16:49:31.607 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-05-08T16:49:31.607 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-05-08T16:49:31.607 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-05-08T16:49:32.139 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-05-08T16:49:32.139 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-05-08T16:49:32.140 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-05-08T16:49:32.140 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-05-08T16:49:32.140 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-05-08T16:49:32.140 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-05-08T16:49:32.140 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-05-08T16:49:32.140 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-05-08T16:49:32.529 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-05-08T16:49:32.529 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-05-08T16:49:32.530 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-05-08T16:49:32.530 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-05-08T16:49:33.049 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-05-08T16:49:33.049 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-05-08T16:49:33.049 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-05-08T16:49:33.049 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:624: TEST_ec_recovery_unfound: ceph pg dump pgs 2024-05-08T16:49:33.469 INFO:tasks.workunit.client.0.smithi172.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-05-08T16:49:33.469 INFO:tasks.workunit.client.0.smithi172.stdout:2.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-05-08T16:49:28.257906+0000 0'0 44:12 [3,1,4,0,2] 3 [3,1,4,0,2] 3 0'0 2024-05-08T16:49:25.965607+0000 0'0 2024-05-08T16:49:25.965607+0000 0 0 periodic scrub scheduled @ 2024-05-10T02:40:06.899180+0000 0 0 2024-05-08T16:49:33.469 INFO:tasks.workunit.client.0.smithi172.stdout:1.3 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-05-08T16:49:25.864105+0000 0'0 45:18 [4,1,2] 4 [4,1,2] 4 0'0 2024-05-08T16:45:47.312563+0000 0'0 2024-05-08T16:45:47.312563+0000 0 0 periodic scrub scheduled @ 2024-05-09T19:19:50.107757+0000 0 0 2024-05-08T16:49:33.469 INFO:tasks.workunit.client.0.smithi172.stdout:1.2 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-05-08T16:49:25.967066+0000 0'0 45:106 [0,4,1] 0 [0,4,1] 0 0'0 2024-05-08T16:45:47.312563+0000 0'0 2024-05-08T16:45:47.312563+0000 0 0 periodic scrub scheduled @ 2024-05-09T21:17:29.777495+0000 0 0 2024-05-08T16:49:33.469 INFO:tasks.workunit.client.0.smithi172.stdout:1.1 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-05-08T16:49:25.562256+0000 0'0 45:18 [4,3,0] 4 [4,3,0] 4 0'0 2024-05-08T16:45:47.312563+0000 0'0 2024-05-08T16:45:47.312563+0000 0 0 periodic scrub scheduled @ 2024-05-09T20:15:21.150504+0000 0 0 2024-05-08T16:49:33.469 INFO:tasks.workunit.client.0.smithi172.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-05-08T16:47:55.359389+0000 0'0 44:69 [1,0,2] 1 [1,0,2] 1 0'0 2024-05-08T16:45:47.312563+0000 0'0 2024-05-08T16:45:47.312563+0000 0 0 periodic scrub scheduled @ 2024-05-09T23:50:28.375129+0000 0 0 2024-05-08T16:49:33.469 INFO:tasks.workunit.client.0.smithi172.stdout: 2024-05-08T16:49:33.469 INFO:tasks.workunit.client.0.smithi172.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-05-08T16:49:33.470 INFO:tasks.workunit.client.0.smithi172.stderr:dumped pgs 2024-05-08T16:49:33.486 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:626: TEST_ec_recovery_unfound: rados_put td/test-erasure-eio pool-jerasure myobject 2024-05-08T16:49:33.486 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:90: rados_put: local dir=td/test-erasure-eio 2024-05-08T16:49:33.486 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:91: rados_put: local poolname=pool-jerasure 2024-05-08T16:49:33.486 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:92: rados_put: local objname=myobject 2024-05-08T16:49:33.486 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-05-08T16:49:33.486 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 AAA 2024-05-08T16:49:33.487 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-05-08T16:49:33.487 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 BBB 2024-05-08T16:49:33.487 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-05-08T16:49:33.487 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 CCCC 2024-05-08T16:49:33.487 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-05-08T16:49:33.487 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 DDDD 2024-05-08T16:49:33.487 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:100: rados_put: rados --pool pool-jerasure put myobject td/test-erasure-eio/ORIGINAL 2024-05-08T16:49:33.678 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:628: TEST_ec_recovery_unfound: get_osds pool-jerasure myobject 2024-05-08T16:49:33.679 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2024-05-08T16:49:33.679 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=myobject 2024-05-08T16:49:33.679 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure myobject 2024-05-08T16:49:33.679 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-05-08T16:49:34.134 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-05-08T16:49:34.135 INFO:tasks.workunit.client.0.smithi172.stderr:1 2024-05-08T16:49:34.135 INFO:tasks.workunit.client.0.smithi172.stderr:4 2024-05-08T16:49:34.135 INFO:tasks.workunit.client.0.smithi172.stderr:0 2024-05-08T16:49:34.135 INFO:tasks.workunit.client.0.smithi172.stderr:2' 2024-05-08T16:49:34.135 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 4 0 2 2024-05-08T16:49:34.135 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:628: TEST_ec_recovery_unfound: initial_osds=('3' '1' '4' '0' '2') 2024-05-08T16:49:34.135 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:628: TEST_ec_recovery_unfound: local -a initial_osds 2024-05-08T16:49:34.135 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:629: TEST_ec_recovery_unfound: local last_osd=2 2024-05-08T16:49:34.135 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:630: TEST_ec_recovery_unfound: kill_daemons td/test-erasure-eio TERM osd.2 2024-05-08T16:49:34.136 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-08T16:49:34.136 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-08T16:49:34.136 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-08T16:49:34.136 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-08T16:49:34.136 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-08T16:49:34.445 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-08T16:49:34.445 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:631: TEST_ec_recovery_unfound: ceph osd down 2 2024-05-08T16:49:35.529 INFO:tasks.workunit.client.0.smithi172.stderr:osd.2 is already down. 2024-05-08T16:49:35.545 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:632: TEST_ec_recovery_unfound: ceph osd out 2 2024-05-08T16:49:36.431 INFO:tasks.workunit.client.0.smithi172.stderr:osd.2 is already out. 2024-05-08T16:49:36.449 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:634: TEST_ec_recovery_unfound: ceph pg dump pgs 2024-05-08T16:49:36.848 INFO:tasks.workunit.client.0.smithi172.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-05-08T16:49:36.848 INFO:tasks.workunit.client.0.smithi172.stdout:2.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-05-08T16:49:28.257906+0000 0'0 45:15 [3,1,4,0,2] 3 [3,1,4,0,2] 3 0'0 2024-05-08T16:49:25.965607+0000 0'0 2024-05-08T16:49:25.965607+0000 0 0 periodic scrub scheduled @ 2024-05-10T03:56:04.764441+0000 0 0 2024-05-08T16:49:36.848 INFO:tasks.workunit.client.0.smithi172.stdout:1.3 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-05-08T16:49:25.864105+0000 0'0 45:18 [4,1,2] 4 [4,1,2] 4 0'0 2024-05-08T16:45:47.312563+0000 0'0 2024-05-08T16:45:47.312563+0000 0 0 periodic scrub scheduled @ 2024-05-09T19:19:50.107757+0000 0 0 2024-05-08T16:49:36.848 INFO:tasks.workunit.client.0.smithi172.stdout:1.2 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-05-08T16:49:25.967066+0000 0'0 45:106 [0,4,1] 0 [0,4,1] 0 0'0 2024-05-08T16:45:47.312563+0000 0'0 2024-05-08T16:45:47.312563+0000 0 0 periodic scrub scheduled @ 2024-05-09T21:17:29.777495+0000 0 0 2024-05-08T16:49:36.848 INFO:tasks.workunit.client.0.smithi172.stdout:1.1 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-05-08T16:49:25.562256+0000 0'0 45:18 [4,3,0] 4 [4,3,0] 4 0'0 2024-05-08T16:45:47.312563+0000 0'0 2024-05-08T16:45:47.312563+0000 0 0 periodic scrub scheduled @ 2024-05-09T20:15:21.150504+0000 0 0 2024-05-08T16:49:36.848 INFO:tasks.workunit.client.0.smithi172.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-05-08T16:47:55.359389+0000 0'0 45:71 [1,0,2] 1 [1,0,2] 1 0'0 2024-05-08T16:45:47.312563+0000 0'0 2024-05-08T16:45:47.312563+0000 0 0 periodic scrub scheduled @ 2024-05-09T23:50:28.375129+0000 0 0 2024-05-08T16:49:36.848 INFO:tasks.workunit.client.0.smithi172.stdout: 2024-05-08T16:49:36.848 INFO:tasks.workunit.client.0.smithi172.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-05-08T16:49:36.848 INFO:tasks.workunit.client.0.smithi172.stderr:dumped pgs 2024-05-08T16:49:36.865 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:636: TEST_ec_recovery_unfound: dd if=/dev/urandom of=td/test-erasure-eio/ORIGINAL bs=1024 count=4 2024-05-08T16:49:36.866 INFO:tasks.workunit.client.0.smithi172.stderr:4+0 records in 2024-05-08T16:49:36.866 INFO:tasks.workunit.client.0.smithi172.stderr:4+0 records out 2024-05-08T16:49:36.866 INFO:tasks.workunit.client.0.smithi172.stderr:4096 bytes (4.1 kB, 4.0 KiB) copied, 0.000184487 s, 22.2 MB/s 2024-05-08T16:49:36.866 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: seq 1 100 2024-05-08T16:49:36.867 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:49:36.867 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj1 td/test-erasure-eio/ORIGINAL 2024-05-08T16:49:37.069 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:49:37.069 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj2 td/test-erasure-eio/ORIGINAL 2024-05-08T16:49:37.764 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:49:37.764 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj3 td/test-erasure-eio/ORIGINAL 2024-05-08T16:49:38.242 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:49:38.242 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj4 td/test-erasure-eio/ORIGINAL 2024-05-08T16:49:38.518 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:49:38.518 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj5 td/test-erasure-eio/ORIGINAL 2024-05-08T16:49:38.896 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:49:38.896 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj6 td/test-erasure-eio/ORIGINAL 2024-05-08T16:49:39.247 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:49:39.247 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj7 td/test-erasure-eio/ORIGINAL 2024-05-08T16:49:39.548 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:49:39.548 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj8 td/test-erasure-eio/ORIGINAL 2024-05-08T16:49:39.850 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:49:39.850 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj9 td/test-erasure-eio/ORIGINAL 2024-05-08T16:49:40.186 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:49:40.186 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj10 td/test-erasure-eio/ORIGINAL 2024-05-08T16:49:40.481 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:49:40.481 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj11 td/test-erasure-eio/ORIGINAL 2024-05-08T16:49:40.790 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:49:40.790 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj12 td/test-erasure-eio/ORIGINAL 2024-05-08T16:49:41.107 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:49:41.107 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj13 td/test-erasure-eio/ORIGINAL 2024-05-08T16:49:41.417 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:49:41.417 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj14 td/test-erasure-eio/ORIGINAL 2024-05-08T16:49:41.827 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:49:41.828 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj15 td/test-erasure-eio/ORIGINAL 2024-05-08T16:49:42.147 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:49:42.147 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj16 td/test-erasure-eio/ORIGINAL 2024-05-08T16:49:42.447 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:49:42.447 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj17 td/test-erasure-eio/ORIGINAL 2024-05-08T16:49:42.775 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:49:42.776 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj18 td/test-erasure-eio/ORIGINAL 2024-05-08T16:49:43.051 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:49:43.051 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj19 td/test-erasure-eio/ORIGINAL 2024-05-08T16:49:43.378 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:49:43.379 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj20 td/test-erasure-eio/ORIGINAL 2024-05-08T16:49:43.680 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:49:43.680 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj21 td/test-erasure-eio/ORIGINAL 2024-05-08T16:49:43.982 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:49:43.982 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj22 td/test-erasure-eio/ORIGINAL 2024-05-08T16:49:44.350 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:49:44.350 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj23 td/test-erasure-eio/ORIGINAL 2024-05-08T16:49:44.654 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:49:44.654 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj24 td/test-erasure-eio/ORIGINAL 2024-05-08T16:49:44.979 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:49:44.979 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj25 td/test-erasure-eio/ORIGINAL 2024-05-08T16:49:45.288 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:49:45.288 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj26 td/test-erasure-eio/ORIGINAL 2024-05-08T16:49:45.581 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:49:45.581 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj27 td/test-erasure-eio/ORIGINAL 2024-05-08T16:49:45.909 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:49:45.909 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj28 td/test-erasure-eio/ORIGINAL 2024-05-08T16:49:46.261 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:49:46.262 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj29 td/test-erasure-eio/ORIGINAL 2024-05-08T16:49:46.580 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:49:46.580 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj30 td/test-erasure-eio/ORIGINAL 2024-05-08T16:49:46.915 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:49:46.915 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj31 td/test-erasure-eio/ORIGINAL 2024-05-08T16:49:47.208 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:49:47.209 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj32 td/test-erasure-eio/ORIGINAL 2024-05-08T16:49:47.551 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:49:47.551 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj33 td/test-erasure-eio/ORIGINAL 2024-05-08T16:49:47.852 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:49:47.852 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj34 td/test-erasure-eio/ORIGINAL 2024-05-08T16:49:48.179 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:49:48.179 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj35 td/test-erasure-eio/ORIGINAL 2024-05-08T16:49:48.498 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:49:48.498 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj36 td/test-erasure-eio/ORIGINAL 2024-05-08T16:49:48.791 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:49:48.791 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj37 td/test-erasure-eio/ORIGINAL 2024-05-08T16:49:49.102 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:49:49.102 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj38 td/test-erasure-eio/ORIGINAL 2024-05-08T16:49:49.369 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:49:49.370 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj39 td/test-erasure-eio/ORIGINAL 2024-05-08T16:49:49.689 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:49:49.689 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj40 td/test-erasure-eio/ORIGINAL 2024-05-08T16:49:49.990 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:49:49.990 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj41 td/test-erasure-eio/ORIGINAL 2024-05-08T16:49:50.350 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:49:50.350 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj42 td/test-erasure-eio/ORIGINAL 2024-05-08T16:49:50.661 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:49:50.662 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj43 td/test-erasure-eio/ORIGINAL 2024-05-08T16:49:50.928 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:49:50.928 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj44 td/test-erasure-eio/ORIGINAL 2024-05-08T16:49:51.238 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:49:51.238 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj45 td/test-erasure-eio/ORIGINAL 2024-05-08T16:49:51.523 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:49:51.524 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj46 td/test-erasure-eio/ORIGINAL 2024-05-08T16:49:51.833 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:49:51.833 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj47 td/test-erasure-eio/ORIGINAL 2024-05-08T16:49:52.160 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:49:52.161 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj48 td/test-erasure-eio/ORIGINAL 2024-05-08T16:49:52.437 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:49:52.437 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj49 td/test-erasure-eio/ORIGINAL 2024-05-08T16:49:52.746 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:49:52.747 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj50 td/test-erasure-eio/ORIGINAL 2024-05-08T16:49:53.031 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:49:53.032 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj51 td/test-erasure-eio/ORIGINAL 2024-05-08T16:49:53.360 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:49:53.360 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj52 td/test-erasure-eio/ORIGINAL 2024-05-08T16:49:53.694 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:49:53.694 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj53 td/test-erasure-eio/ORIGINAL 2024-05-08T16:49:54.031 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:49:54.031 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj54 td/test-erasure-eio/ORIGINAL 2024-05-08T16:49:54.331 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:49:54.331 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj55 td/test-erasure-eio/ORIGINAL 2024-05-08T16:49:54.678 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:49:54.679 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj56 td/test-erasure-eio/ORIGINAL 2024-05-08T16:49:54.943 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:49:54.943 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj57 td/test-erasure-eio/ORIGINAL 2024-05-08T16:49:55.262 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:49:55.262 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj58 td/test-erasure-eio/ORIGINAL 2024-05-08T16:49:55.555 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:49:55.555 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj59 td/test-erasure-eio/ORIGINAL 2024-05-08T16:49:55.840 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:49:55.841 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj60 td/test-erasure-eio/ORIGINAL 2024-05-08T16:49:56.174 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:49:56.174 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj61 td/test-erasure-eio/ORIGINAL 2024-05-08T16:49:56.484 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:49:56.484 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj62 td/test-erasure-eio/ORIGINAL 2024-05-08T16:49:56.803 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:49:56.804 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj63 td/test-erasure-eio/ORIGINAL 2024-05-08T16:49:57.105 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:49:57.105 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj64 td/test-erasure-eio/ORIGINAL 2024-05-08T16:49:57.398 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:49:57.398 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj65 td/test-erasure-eio/ORIGINAL 2024-05-08T16:49:57.724 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:49:57.724 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj66 td/test-erasure-eio/ORIGINAL 2024-05-08T16:49:58.043 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:49:58.043 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj67 td/test-erasure-eio/ORIGINAL 2024-05-08T16:49:58.371 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:49:58.371 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj68 td/test-erasure-eio/ORIGINAL 2024-05-08T16:49:58.680 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:49:58.680 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj69 td/test-erasure-eio/ORIGINAL 2024-05-08T16:49:58.975 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:49:58.975 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj70 td/test-erasure-eio/ORIGINAL 2024-05-08T16:49:59.300 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:49:59.301 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj71 td/test-erasure-eio/ORIGINAL 2024-05-08T16:49:59.544 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:49:59.544 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj72 td/test-erasure-eio/ORIGINAL 2024-05-08T16:49:59.912 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:49:59.912 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj73 td/test-erasure-eio/ORIGINAL 2024-05-08T16:50:00.232 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:50:00.232 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj74 td/test-erasure-eio/ORIGINAL 2024-05-08T16:50:00.574 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:50:00.574 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj75 td/test-erasure-eio/ORIGINAL 2024-05-08T16:50:00.927 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:50:00.927 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj76 td/test-erasure-eio/ORIGINAL 2024-05-08T16:50:01.178 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:50:01.178 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj77 td/test-erasure-eio/ORIGINAL 2024-05-08T16:50:01.462 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:50:01.463 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj78 td/test-erasure-eio/ORIGINAL 2024-05-08T16:50:01.747 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:50:01.747 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj79 td/test-erasure-eio/ORIGINAL 2024-05-08T16:50:02.040 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:50:02.040 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj80 td/test-erasure-eio/ORIGINAL 2024-05-08T16:50:02.593 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:50:02.594 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj81 td/test-erasure-eio/ORIGINAL 2024-05-08T16:50:02.911 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:50:02.911 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj82 td/test-erasure-eio/ORIGINAL 2024-05-08T16:50:03.229 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:50:03.230 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj83 td/test-erasure-eio/ORIGINAL 2024-05-08T16:50:03.540 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:50:03.540 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj84 td/test-erasure-eio/ORIGINAL 2024-05-08T16:50:03.841 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:50:03.841 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj85 td/test-erasure-eio/ORIGINAL 2024-05-08T16:50:04.144 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:50:04.144 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj86 td/test-erasure-eio/ORIGINAL 2024-05-08T16:50:04.495 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:50:04.495 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj87 td/test-erasure-eio/ORIGINAL 2024-05-08T16:50:04.805 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:50:04.805 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj88 td/test-erasure-eio/ORIGINAL 2024-05-08T16:50:05.123 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:50:05.123 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj89 td/test-erasure-eio/ORIGINAL 2024-05-08T16:50:05.417 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:50:05.417 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj90 td/test-erasure-eio/ORIGINAL 2024-05-08T16:50:05.727 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:50:05.727 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj91 td/test-erasure-eio/ORIGINAL 2024-05-08T16:50:06.046 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:50:06.046 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj92 td/test-erasure-eio/ORIGINAL 2024-05-08T16:50:06.338 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:50:06.338 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj93 td/test-erasure-eio/ORIGINAL 2024-05-08T16:50:06.657 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:50:06.657 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj94 td/test-erasure-eio/ORIGINAL 2024-05-08T16:50:06.968 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:50:06.968 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj95 td/test-erasure-eio/ORIGINAL 2024-05-08T16:50:07.313 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:50:07.313 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj96 td/test-erasure-eio/ORIGINAL 2024-05-08T16:50:07.599 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:50:07.599 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj97 td/test-erasure-eio/ORIGINAL 2024-05-08T16:50:07.923 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:50:07.923 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj98 td/test-erasure-eio/ORIGINAL 2024-05-08T16:50:08.276 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:50:08.276 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj99 td/test-erasure-eio/ORIGINAL 2024-05-08T16:50:08.603 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:637: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:50:08.603 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:639: TEST_ec_recovery_unfound: rados --pool pool-jerasure put obj100 td/test-erasure-eio/ORIGINAL 2024-05-08T16:50:08.882 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:642: TEST_ec_recovery_unfound: inject_eio ec data pool-jerasure obj75 td/test-erasure-eio 0 2024-05-08T16:50:08.882 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2394: inject_eio: local pooltype=ec 2024-05-08T16:50:08.882 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2395: inject_eio: shift 2024-05-08T16:50:08.882 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2396: inject_eio: local which=data 2024-05-08T16:50:08.882 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2397: inject_eio: shift 2024-05-08T16:50:08.882 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2398: inject_eio: local poolname=pool-jerasure 2024-05-08T16:50:08.882 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2399: inject_eio: shift 2024-05-08T16:50:08.882 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2400: inject_eio: local objname=obj75 2024-05-08T16:50:08.882 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2401: inject_eio: shift 2024-05-08T16:50:08.882 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2402: inject_eio: local dir=td/test-erasure-eio 2024-05-08T16:50:08.882 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2403: inject_eio: shift 2024-05-08T16:50:08.883 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2404: inject_eio: local shard_id=0 2024-05-08T16:50:08.883 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2405: inject_eio: shift 2024-05-08T16:50:08.883 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: get_osds pool-jerasure obj75 2024-05-08T16:50:08.883 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2024-05-08T16:50:08.883 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj75 2024-05-08T16:50:08.883 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure obj75 2024-05-08T16:50:08.883 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-05-08T16:50:09.305 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-05-08T16:50:09.306 INFO:tasks.workunit.client.0.smithi172.stderr:1 2024-05-08T16:50:09.306 INFO:tasks.workunit.client.0.smithi172.stderr:4 2024-05-08T16:50:09.306 INFO:tasks.workunit.client.0.smithi172.stderr:0 2024-05-08T16:50:09.306 INFO:tasks.workunit.client.0.smithi172.stderr:2147483647' 2024-05-08T16:50:09.306 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 4 0 2147483647 2024-05-08T16:50:09.306 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: initial_osds=('3' '1' '4' '0' '2147483647') 2024-05-08T16:50:09.306 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: local -a initial_osds 2024-05-08T16:50:09.306 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2408: inject_eio: local osd_id=3 2024-05-08T16:50:09.306 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2409: inject_eio: '[' ec '!=' ec ']' 2024-05-08T16:50:09.307 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: cat td/test-erasure-eio/3/type 2024-05-08T16:50:09.307 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: type=bluestore 2024-05-08T16:50:09.307 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: set_config osd 3 bluestore_debug_inject_read_err true 2024-05-08T16:50:09.308 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-05-08T16:50:09.308 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-05-08T16:50:09.308 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-05-08T16:50:09.308 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-05-08T16:50:09.309 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-05-08T16:50:09.309 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-05-08T16:50:09.309 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-05-08T16:50:09.309 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-05-08T16:50:09.309 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-08T16:50:09.309 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T16:50:09.309 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81110 2024-05-08T16:50:09.309 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.81110/ceph-osd.3.asok 2024-05-08T16:50:09.310 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.81110/ceph-osd.3.asok config set bluestore_debug_inject_read_err true 2024-05-08T16:50:09.416 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-05-08T16:50:09.416 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local loop=0 2024-05-08T16:50:09.417 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: grep -q Invalid 2024-05-08T16:50:09.418 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: get_asok_path osd.3 2024-05-08T16:50:09.418 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-05-08T16:50:09.418 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-05-08T16:50:09.419 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-08T16:50:09.419 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T16:50:09.419 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81110 2024-05-08T16:50:09.419 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.81110/ceph-osd.3.asok 2024-05-08T16:50:09.419 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: CEPH_ARGS= 2024-05-08T16:50:09.419 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: ceph --admin-daemon /tmp/ceph-asok.81110/ceph-osd.3.asok injectdataerr pool-jerasure obj75 0 2024-05-08T16:50:09.538 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:643: TEST_ec_recovery_unfound: inject_eio ec data pool-jerasure obj75 td/test-erasure-eio 1 2024-05-08T16:50:09.538 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2394: inject_eio: local pooltype=ec 2024-05-08T16:50:09.538 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2395: inject_eio: shift 2024-05-08T16:50:09.538 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2396: inject_eio: local which=data 2024-05-08T16:50:09.538 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2397: inject_eio: shift 2024-05-08T16:50:09.538 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2398: inject_eio: local poolname=pool-jerasure 2024-05-08T16:50:09.538 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2399: inject_eio: shift 2024-05-08T16:50:09.538 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2400: inject_eio: local objname=obj75 2024-05-08T16:50:09.538 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2401: inject_eio: shift 2024-05-08T16:50:09.538 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2402: inject_eio: local dir=td/test-erasure-eio 2024-05-08T16:50:09.538 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2403: inject_eio: shift 2024-05-08T16:50:09.538 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2404: inject_eio: local shard_id=1 2024-05-08T16:50:09.539 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2405: inject_eio: shift 2024-05-08T16:50:09.539 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: get_osds pool-jerasure obj75 2024-05-08T16:50:09.539 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2024-05-08T16:50:09.539 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj75 2024-05-08T16:50:09.540 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure obj75 2024-05-08T16:50:09.540 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-05-08T16:50:09.981 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-05-08T16:50:09.981 INFO:tasks.workunit.client.0.smithi172.stderr:1 2024-05-08T16:50:09.981 INFO:tasks.workunit.client.0.smithi172.stderr:4 2024-05-08T16:50:09.981 INFO:tasks.workunit.client.0.smithi172.stderr:0 2024-05-08T16:50:09.981 INFO:tasks.workunit.client.0.smithi172.stderr:2147483647' 2024-05-08T16:50:09.981 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 4 0 2147483647 2024-05-08T16:50:09.981 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: initial_osds=('3' '1' '4' '0' '2147483647') 2024-05-08T16:50:09.981 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: local -a initial_osds 2024-05-08T16:50:09.981 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2408: inject_eio: local osd_id=1 2024-05-08T16:50:09.981 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2409: inject_eio: '[' ec '!=' ec ']' 2024-05-08T16:50:09.982 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: cat td/test-erasure-eio/1/type 2024-05-08T16:50:09.982 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: type=bluestore 2024-05-08T16:50:09.983 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2024-05-08T16:50:09.983 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-05-08T16:50:09.983 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=1 2024-05-08T16:50:09.983 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-05-08T16:50:09.983 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-05-08T16:50:09.983 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-05-08T16:50:09.983 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.1 2024-05-08T16:50:09.984 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-05-08T16:50:09.984 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-05-08T16:50:09.984 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-08T16:50:09.984 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T16:50:09.985 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81110 2024-05-08T16:50:09.985 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.81110/ceph-osd.1.asok 2024-05-08T16:50:09.985 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.81110/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2024-05-08T16:50:10.105 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-05-08T16:50:10.106 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local loop=0 2024-05-08T16:50:10.106 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: grep -q Invalid 2024-05-08T16:50:10.107 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: get_asok_path osd.1 2024-05-08T16:50:10.107 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-05-08T16:50:10.107 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-05-08T16:50:10.107 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-08T16:50:10.107 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T16:50:10.107 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81110 2024-05-08T16:50:10.108 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.81110/ceph-osd.1.asok 2024-05-08T16:50:10.108 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: CEPH_ARGS= 2024-05-08T16:50:10.108 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: ceph --admin-daemon /tmp/ceph-asok.81110/ceph-osd.1.asok injectdataerr pool-jerasure obj75 1 2024-05-08T16:50:10.220 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:645: TEST_ec_recovery_unfound: activate_osd td/test-erasure-eio 2 2024-05-08T16:50:10.220 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/test-erasure-eio 2024-05-08T16:50:10.221 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-05-08T16:50:10.221 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-05-08T16:50:10.221 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-05-08T16:50:10.221 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/test-erasure-eio/2 2024-05-08T16:50:10.221 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=4457b0f1-028b-482b-8ee3-7ee05861e6cc --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-05-08T16:50:10.221 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-08T16:50:10.221 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-05-08T16:50:10.221 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-08T16:50:10.221 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/2' 2024-05-08T16:50:10.221 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/2/journal' 2024-05-08T16:50:10.221 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-05-08T16:50:10.221 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-05-08T16:50:10.221 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-05-08T16:50:10.221 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-05-08T16:50:10.221 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-08T16:50:10.221 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-08T16:50:10.222 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-08T16:50:10.222 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T16:50:10.222 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81110 2024-05-08T16:50:10.223 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.81110/$cluster-$name.asok' 2024-05-08T16:50:10.223 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.81110/$cluster-$name.asok' 2024-05-08T16:50:10.223 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-05-08T16:50:10.223 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-05-08T16:50:10.223 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-05-08T16:50:10.223 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-08T16:50:10.223 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-08T16:50:10.223 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-08T16:50:10.223 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-08T16:50:10.223 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-05-08T16:50:10.223 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-05-08T16:50:10.223 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/test-erasure-eio/2 2024-05-08T16:50:10.224 INFO:tasks.workunit.client.0.smithi172.stdout:start osd.2 2024-05-08T16:50:10.224 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-05-08T16:50:10.224 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=4457b0f1-028b-482b-8ee3-7ee05861e6cc --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81110/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-05-08T16:50:10.224 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/test-erasure-eio/2/whoami 2024-05-08T16:50:10.225 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-05-08T16:50:10.225 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-05-08T16:50:10.226 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-05-08T16:50:10.226 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-05-08T16:50:10.248 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:50:10.249+0000 7f2db772f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:50:10.251 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:50:10.253+0000 7f2db772f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:50:10.252 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:50:10.253+0000 7f2db772f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:50:10.661 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-05-08T16:50:10.662 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-08T16:50:10.662 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-05-08T16:50:10.662 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-08T16:50:10.662 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-08T16:50:10.662 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:50:10.662 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-08T16:50:10.662 INFO:tasks.workunit.client.0.smithi172.stdout:0 2024-05-08T16:50:10.663 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:50:10.663 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T16:50:10.803 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:50:10.805+0000 7f2db772f7c0 -1 Falling back to public interface 2024-05-08T16:50:11.096 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:50:12.098 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:50:12.098 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:50:12.098 INFO:tasks.workunit.client.0.smithi172.stdout:1 2024-05-08T16:50:12.099 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-08T16:50:12.099 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:50:12.099 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T16:50:12.557 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:50:13.558 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:50:13.559 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:50:13.559 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-08T16:50:13.559 INFO:tasks.workunit.client.0.smithi172.stdout:2 2024-05-08T16:50:13.559 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:50:13.560 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T16:50:13.975 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:50:14.524 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:50:14.525+0000 7f2db772f7c0 -1 osd.2 45 log_to_monitors true 2024-05-08T16:50:14.977 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:50:14.977 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:50:14.977 INFO:tasks.workunit.client.0.smithi172.stdout:3 2024-05-08T16:50:14.978 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-05-08T16:50:14.978 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:50:14.978 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T16:50:15.590 INFO:tasks.workunit.client.0.smithi172.stdout:osd.2 up out weight 0 up_from 53 up_thru 19 down_at 46 last_clean_interval [19,45) [v2:127.0.0.1:6818/2352396965,v1:127.0.0.1:6819/2352396965] [v2:127.0.0.1:6820/2352396965,v1:127.0.0.1:6821/2352396965] exists,up da1c5da2-b226-476c-8ffd-001130f7d176 2024-05-08T16:50:15.591 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-08T16:50:15.591 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-08T16:50:15.591 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-08T16:50:15.591 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:646: TEST_ec_recovery_unfound: ceph osd in 2 2024-05-08T16:50:16.561 INFO:tasks.workunit.client.0.smithi172.stderr:osd.2 is already in. 2024-05-08T16:50:16.583 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:648: TEST_ec_recovery_unfound: sleep 15 2024-05-08T16:50:31.586 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:650: TEST_ec_recovery_unfound: seq 1 100 2024-05-08T16:50:31.587 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:650: TEST_ec_recovery_unfound: for tmp in $(seq 1 100) 2024-05-08T16:50:31.588 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:651: TEST_ec_recovery_unfound: get_state 2.0 2024-05-08T16:50:31.588 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:60: get_state: local pgid=2.0 2024-05-08T16:50:31.588 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:61: get_state: local sname=state 2024-05-08T16:50:31.588 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:62: get_state: ceph --format json pg dump pgs 2024-05-08T16:50:31.588 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:63: get_state: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .state' 2024-05-08T16:50:32.002 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:651: TEST_ec_recovery_unfound: state=active+recovery_unfound+degraded 2024-05-08T16:50:32.002 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:652: TEST_ec_recovery_unfound: echo active+recovery_unfound+degraded 2024-05-08T16:50:32.002 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:652: TEST_ec_recovery_unfound: grep recovery_unfound 2024-05-08T16:50:32.003 INFO:tasks.workunit.client.0.smithi172.stdout:active+recovery_unfound+degraded 2024-05-08T16:50:32.003 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:653: TEST_ec_recovery_unfound: '[' 0 = 0 ']' 2024-05-08T16:50:32.004 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:654: TEST_ec_recovery_unfound: break 2024-05-08T16:50:32.004 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:660: TEST_ec_recovery_unfound: ceph pg dump pgs 2024-05-08T16:50:32.369 INFO:tasks.workunit.client.0.smithi172.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-05-08T16:50:32.369 INFO:tasks.workunit.client.0.smithi172.stdout:2.0 101 1 3 0 1 413696 0 0 101 0 101 active+recovery_unfound+degraded 2024-05-08T16:50:22.401967+0000 51'101 58:351 [3,1,4,0,2] 3 [3,1,4,0,2] 3 0'0 2024-05-08T16:49:25.965607+0000 0'0 2024-05-08T16:49:25.965607+0000 0 0 no scrub is scheduled 0 0 2024-05-08T16:50:32.369 INFO:tasks.workunit.client.0.smithi172.stdout:1.3 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-05-08T16:50:18.298588+0000 0'0 58:58 [4,1,2] 4 [4,1,2] 4 0'0 2024-05-08T16:45:47.312563+0000 0'0 2024-05-08T16:45:47.312563+0000 0 0 periodic scrub scheduled @ 2024-05-10T02:29:47.004679+0000 0 0 2024-05-08T16:50:32.369 INFO:tasks.workunit.client.0.smithi172.stdout:1.2 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-05-08T16:49:25.967066+0000 0'0 58:130 [0,4,1] 0 [0,4,1] 0 0'0 2024-05-08T16:45:47.312563+0000 0'0 2024-05-08T16:45:47.312563+0000 0 0 periodic scrub scheduled @ 2024-05-09T21:17:29.777495+0000 0 0 2024-05-08T16:50:32.369 INFO:tasks.workunit.client.0.smithi172.stdout:1.1 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-05-08T16:49:25.562256+0000 0'0 58:44 [4,3,0] 4 [4,3,0] 4 0'0 2024-05-08T16:45:47.312563+0000 0'0 2024-05-08T16:45:47.312563+0000 0 0 periodic scrub scheduled @ 2024-05-09T20:15:21.150504+0000 0 0 2024-05-08T16:50:32.370 INFO:tasks.workunit.client.0.smithi172.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-05-08T16:50:18.363931+0000 0'0 58:109 [1,0,2] 1 [1,0,2] 1 0'0 2024-05-08T16:45:47.312563+0000 0'0 2024-05-08T16:45:47.312563+0000 0 0 periodic scrub scheduled @ 2024-05-10T03:41:04.045111+0000 0 0 2024-05-08T16:50:32.370 INFO:tasks.workunit.client.0.smithi172.stdout: 2024-05-08T16:50:32.370 INFO:tasks.workunit.client.0.smithi172.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-05-08T16:50:32.370 INFO:tasks.workunit.client.0.smithi172.stderr:dumped pgs 2024-05-08T16:50:32.384 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:661: TEST_ec_recovery_unfound: ceph pg 2.0 list_unfound 2024-05-08T16:50:32.532 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2024-05-08T16:50:32.533 INFO:tasks.workunit.client.0.smithi172.stdout: "num_missing": 1, 2024-05-08T16:50:32.533 INFO:tasks.workunit.client.0.smithi172.stdout: "num_unfound": 1, 2024-05-08T16:50:32.533 INFO:tasks.workunit.client.0.smithi172.stdout: "objects": [ 2024-05-08T16:50:32.533 INFO:tasks.workunit.client.0.smithi172.stdout: { 2024-05-08T16:50:32.533 INFO:tasks.workunit.client.0.smithi172.stdout: "oid": { 2024-05-08T16:50:32.533 INFO:tasks.workunit.client.0.smithi172.stdout: "oid": "obj75", 2024-05-08T16:50:32.533 INFO:tasks.workunit.client.0.smithi172.stdout: "key": "", 2024-05-08T16:50:32.533 INFO:tasks.workunit.client.0.smithi172.stdout: "snapid": -2, 2024-05-08T16:50:32.533 INFO:tasks.workunit.client.0.smithi172.stdout: "hash": 3565005477, 2024-05-08T16:50:32.533 INFO:tasks.workunit.client.0.smithi172.stdout: "max": 0, 2024-05-08T16:50:32.533 INFO:tasks.workunit.client.0.smithi172.stdout: "pool": 2, 2024-05-08T16:50:32.533 INFO:tasks.workunit.client.0.smithi172.stdout: "namespace": "" 2024-05-08T16:50:32.534 INFO:tasks.workunit.client.0.smithi172.stdout: }, 2024-05-08T16:50:32.534 INFO:tasks.workunit.client.0.smithi172.stdout: "need": "51'76", 2024-05-08T16:50:32.534 INFO:tasks.workunit.client.0.smithi172.stdout: "have": "0'0", 2024-05-08T16:50:32.534 INFO:tasks.workunit.client.0.smithi172.stdout: "flags": "none", 2024-05-08T16:50:32.534 INFO:tasks.workunit.client.0.smithi172.stdout: "clean_regions": "clean_offsets: [], clean_omap: false, new_object: true", 2024-05-08T16:50:32.534 INFO:tasks.workunit.client.0.smithi172.stdout: "locations": [ 2024-05-08T16:50:32.534 INFO:tasks.workunit.client.0.smithi172.stdout: "0(3)", 2024-05-08T16:50:32.534 INFO:tasks.workunit.client.0.smithi172.stdout: "4(2)" 2024-05-08T16:50:32.534 INFO:tasks.workunit.client.0.smithi172.stdout: ] 2024-05-08T16:50:32.534 INFO:tasks.workunit.client.0.smithi172.stdout: } 2024-05-08T16:50:32.534 INFO:tasks.workunit.client.0.smithi172.stdout: ], 2024-05-08T16:50:32.534 INFO:tasks.workunit.client.0.smithi172.stdout: "state": "NotRecovering", 2024-05-08T16:50:32.534 INFO:tasks.workunit.client.0.smithi172.stdout: "available_might_have_unfound": true, 2024-05-08T16:50:32.534 INFO:tasks.workunit.client.0.smithi172.stdout: "might_have_unfound": [], 2024-05-08T16:50:32.535 INFO:tasks.workunit.client.0.smithi172.stdout: "more": false 2024-05-08T16:50:32.535 INFO:tasks.workunit.client.0.smithi172.stdout:} 2024-05-08T16:50:32.546 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:662: TEST_ec_recovery_unfound: ceph pg 2.0 query 2024-05-08T16:50:32.681 INFO:tasks.workunit.client.0.smithi172.stdout:{ 2024-05-08T16:50:32.681 INFO:tasks.workunit.client.0.smithi172.stdout: "snap_trimq": "[]", 2024-05-08T16:50:32.681 INFO:tasks.workunit.client.0.smithi172.stdout: "snap_trimq_len": 0, 2024-05-08T16:50:32.681 INFO:tasks.workunit.client.0.smithi172.stdout: "state": "active+recovery_unfound+degraded", 2024-05-08T16:50:32.681 INFO:tasks.workunit.client.0.smithi172.stdout: "epoch": 58, 2024-05-08T16:50:32.681 INFO:tasks.workunit.client.0.smithi172.stdout: "up": [ 2024-05-08T16:50:32.681 INFO:tasks.workunit.client.0.smithi172.stdout: 3, 2024-05-08T16:50:32.682 INFO:tasks.workunit.client.0.smithi172.stdout: 1, 2024-05-08T16:50:32.682 INFO:tasks.workunit.client.0.smithi172.stdout: 4, 2024-05-08T16:50:32.682 INFO:tasks.workunit.client.0.smithi172.stdout: 0, 2024-05-08T16:50:32.682 INFO:tasks.workunit.client.0.smithi172.stdout: 2 2024-05-08T16:50:32.682 INFO:tasks.workunit.client.0.smithi172.stdout: ], 2024-05-08T16:50:32.682 INFO:tasks.workunit.client.0.smithi172.stdout: "acting": [ 2024-05-08T16:50:32.682 INFO:tasks.workunit.client.0.smithi172.stdout: 3, 2024-05-08T16:50:32.682 INFO:tasks.workunit.client.0.smithi172.stdout: 1, 2024-05-08T16:50:32.682 INFO:tasks.workunit.client.0.smithi172.stdout: 4, 2024-05-08T16:50:32.682 INFO:tasks.workunit.client.0.smithi172.stdout: 0, 2024-05-08T16:50:32.682 INFO:tasks.workunit.client.0.smithi172.stdout: 2 2024-05-08T16:50:32.682 INFO:tasks.workunit.client.0.smithi172.stdout: ], 2024-05-08T16:50:32.682 INFO:tasks.workunit.client.0.smithi172.stdout: "acting_recovery_backfill": [ 2024-05-08T16:50:32.682 INFO:tasks.workunit.client.0.smithi172.stdout: "0(3)", 2024-05-08T16:50:32.682 INFO:tasks.workunit.client.0.smithi172.stdout: "1(1)", 2024-05-08T16:50:32.683 INFO:tasks.workunit.client.0.smithi172.stdout: "2(4)", 2024-05-08T16:50:32.683 INFO:tasks.workunit.client.0.smithi172.stdout: "3(0)", 2024-05-08T16:50:32.683 INFO:tasks.workunit.client.0.smithi172.stdout: "4(2)" 2024-05-08T16:50:32.683 INFO:tasks.workunit.client.0.smithi172.stdout: ], 2024-05-08T16:50:32.683 INFO:tasks.workunit.client.0.smithi172.stdout: "info": { 2024-05-08T16:50:32.683 INFO:tasks.workunit.client.0.smithi172.stdout: "pgid": "2.0s0", 2024-05-08T16:50:32.683 INFO:tasks.workunit.client.0.smithi172.stdout: "shared": "0", 2024-05-08T16:50:32.683 INFO:tasks.workunit.client.0.smithi172.stdout: "last_update": "51'101", 2024-05-08T16:50:32.683 INFO:tasks.workunit.client.0.smithi172.stdout: "last_complete": "51'75", 2024-05-08T16:50:32.683 INFO:tasks.workunit.client.0.smithi172.stdout: "log_tail": "0'0", 2024-05-08T16:50:32.683 INFO:tasks.workunit.client.0.smithi172.stdout: "last_user_version": 101, 2024-05-08T16:50:32.683 INFO:tasks.workunit.client.0.smithi172.stdout: "last_backfill": "MAX", 2024-05-08T16:50:32.683 INFO:tasks.workunit.client.0.smithi172.stdout: "purged_snaps": [], 2024-05-08T16:50:32.683 INFO:tasks.workunit.client.0.smithi172.stdout: "history": { 2024-05-08T16:50:32.683 INFO:tasks.workunit.client.0.smithi172.stdout: "epoch_created": 43, 2024-05-08T16:50:32.684 INFO:tasks.workunit.client.0.smithi172.stdout: "epoch_pool_created": 43, 2024-05-08T16:50:32.684 INFO:tasks.workunit.client.0.smithi172.stdout: "last_epoch_started": 58, 2024-05-08T16:50:32.684 INFO:tasks.workunit.client.0.smithi172.stdout: "last_interval_started": 55, 2024-05-08T16:50:32.684 INFO:tasks.workunit.client.0.smithi172.stdout: "last_epoch_clean": 44, 2024-05-08T16:50:32.684 INFO:tasks.workunit.client.0.smithi172.stdout: "last_interval_clean": 43, 2024-05-08T16:50:32.684 INFO:tasks.workunit.client.0.smithi172.stdout: "last_epoch_split": 0, 2024-05-08T16:50:32.684 INFO:tasks.workunit.client.0.smithi172.stdout: "last_epoch_marked_full": 0, 2024-05-08T16:50:32.684 INFO:tasks.workunit.client.0.smithi172.stdout: "same_up_since": 55, 2024-05-08T16:50:32.684 INFO:tasks.workunit.client.0.smithi172.stdout: "same_interval_since": 55, 2024-05-08T16:50:32.684 INFO:tasks.workunit.client.0.smithi172.stdout: "same_primary_since": 43, 2024-05-08T16:50:32.684 INFO:tasks.workunit.client.0.smithi172.stdout: "last_scrub": "0'0", 2024-05-08T16:50:32.684 INFO:tasks.workunit.client.0.smithi172.stdout: "last_scrub_stamp": "2024-05-08T16:49:25.965607+0000", 2024-05-08T16:50:32.684 INFO:tasks.workunit.client.0.smithi172.stdout: "last_deep_scrub": "0'0", 2024-05-08T16:50:32.684 INFO:tasks.workunit.client.0.smithi172.stdout: "last_deep_scrub_stamp": "2024-05-08T16:49:25.965607+0000", 2024-05-08T16:50:32.684 INFO:tasks.workunit.client.0.smithi172.stdout: "last_clean_scrub_stamp": "2024-05-08T16:49:25.965607+0000", 2024-05-08T16:50:32.685 INFO:tasks.workunit.client.0.smithi172.stdout: "prior_readable_until_ub": 0 2024-05-08T16:50:32.685 INFO:tasks.workunit.client.0.smithi172.stdout: }, 2024-05-08T16:50:32.685 INFO:tasks.workunit.client.0.smithi172.stdout: "stats": { 2024-05-08T16:50:32.685 INFO:tasks.workunit.client.0.smithi172.stdout: "version": "51'101", 2024-05-08T16:50:32.685 INFO:tasks.workunit.client.0.smithi172.stdout: "reported_seq": 352, 2024-05-08T16:50:32.685 INFO:tasks.workunit.client.0.smithi172.stdout: "reported_epoch": 58, 2024-05-08T16:50:32.685 INFO:tasks.workunit.client.0.smithi172.stdout: "state": "active+recovery_unfound+degraded", 2024-05-08T16:50:32.685 INFO:tasks.workunit.client.0.smithi172.stdout: "last_fresh": "2024-05-08T16:50:22.489798+0000", 2024-05-08T16:50:32.685 INFO:tasks.workunit.client.0.smithi172.stdout: "last_change": "2024-05-08T16:50:22.401967+0000", 2024-05-08T16:50:32.685 INFO:tasks.workunit.client.0.smithi172.stdout: "last_active": "2024-05-08T16:50:22.489798+0000", 2024-05-08T16:50:32.685 INFO:tasks.workunit.client.0.smithi172.stdout: "last_peered": "2024-05-08T16:50:22.489798+0000", 2024-05-08T16:50:32.685 INFO:tasks.workunit.client.0.smithi172.stdout: "last_clean": "2024-05-08T16:49:33.679662+0000", 2024-05-08T16:50:32.685 INFO:tasks.workunit.client.0.smithi172.stdout: "last_became_active": "2024-05-08T16:50:18.862714+0000", 2024-05-08T16:50:32.686 INFO:tasks.workunit.client.0.smithi172.stdout: "last_became_peered": "2024-05-08T16:50:18.862714+0000", 2024-05-08T16:50:32.686 INFO:tasks.workunit.client.0.smithi172.stdout: "last_unstale": "2024-05-08T16:50:22.489798+0000", 2024-05-08T16:50:32.686 INFO:tasks.workunit.client.0.smithi172.stdout: "last_undegraded": "2024-05-08T16:50:17.988450+0000", 2024-05-08T16:50:32.686 INFO:tasks.workunit.client.0.smithi172.stdout: "last_fullsized": "2024-05-08T16:50:22.489798+0000", 2024-05-08T16:50:32.686 INFO:tasks.workunit.client.0.smithi172.stdout: "mapping_epoch": 55, 2024-05-08T16:50:32.686 INFO:tasks.workunit.client.0.smithi172.stdout: "log_start": "0'0", 2024-05-08T16:50:32.686 INFO:tasks.workunit.client.0.smithi172.stdout: "ondisk_log_start": "0'0", 2024-05-08T16:50:32.686 INFO:tasks.workunit.client.0.smithi172.stdout: "created": 43, 2024-05-08T16:50:32.686 INFO:tasks.workunit.client.0.smithi172.stdout: "last_epoch_clean": 44, 2024-05-08T16:50:32.686 INFO:tasks.workunit.client.0.smithi172.stdout: "parent": "0.0", 2024-05-08T16:50:32.686 INFO:tasks.workunit.client.0.smithi172.stdout: "parent_split_bits": 0, 2024-05-08T16:50:32.686 INFO:tasks.workunit.client.0.smithi172.stdout: "last_scrub": "0'0", 2024-05-08T16:50:32.686 INFO:tasks.workunit.client.0.smithi172.stdout: "last_scrub_stamp": "2024-05-08T16:49:25.965607+0000", 2024-05-08T16:50:32.687 INFO:tasks.workunit.client.0.smithi172.stdout: "last_deep_scrub": "0'0", 2024-05-08T16:50:32.687 INFO:tasks.workunit.client.0.smithi172.stdout: "last_deep_scrub_stamp": "2024-05-08T16:49:25.965607+0000", 2024-05-08T16:50:32.687 INFO:tasks.workunit.client.0.smithi172.stdout: "last_clean_scrub_stamp": "2024-05-08T16:49:25.965607+0000", 2024-05-08T16:50:32.687 INFO:tasks.workunit.client.0.smithi172.stdout: "objects_scrubbed": 0, 2024-05-08T16:50:32.687 INFO:tasks.workunit.client.0.smithi172.stdout: "log_size": 101, 2024-05-08T16:50:32.687 INFO:tasks.workunit.client.0.smithi172.stdout: "log_dups_size": 0, 2024-05-08T16:50:32.687 INFO:tasks.workunit.client.0.smithi172.stdout: "ondisk_log_size": 101, 2024-05-08T16:50:32.687 INFO:tasks.workunit.client.0.smithi172.stdout: "stats_invalid": false, 2024-05-08T16:50:32.687 INFO:tasks.workunit.client.0.smithi172.stdout: "dirty_stats_invalid": false, 2024-05-08T16:50:32.687 INFO:tasks.workunit.client.0.smithi172.stdout: "omap_stats_invalid": false, 2024-05-08T16:50:32.687 INFO:tasks.workunit.client.0.smithi172.stdout: "hitset_stats_invalid": false, 2024-05-08T16:50:32.687 INFO:tasks.workunit.client.0.smithi172.stdout: "hitset_bytes_stats_invalid": false, 2024-05-08T16:50:32.687 INFO:tasks.workunit.client.0.smithi172.stdout: "pin_stats_invalid": false, 2024-05-08T16:50:32.687 INFO:tasks.workunit.client.0.smithi172.stdout: "manifest_stats_invalid": false, 2024-05-08T16:50:32.688 INFO:tasks.workunit.client.0.smithi172.stdout: "snaptrimq_len": 0, 2024-05-08T16:50:32.688 INFO:tasks.workunit.client.0.smithi172.stdout: "last_scrub_duration": 0, 2024-05-08T16:50:32.688 INFO:tasks.workunit.client.0.smithi172.stdout: "scrub_schedule": "no scrub is scheduled", 2024-05-08T16:50:32.688 INFO:tasks.workunit.client.0.smithi172.stdout: "scrub_duration": 0, 2024-05-08T16:50:32.688 INFO:tasks.workunit.client.0.smithi172.stdout: "objects_trimmed": 0, 2024-05-08T16:50:32.688 INFO:tasks.workunit.client.0.smithi172.stdout: "snaptrim_duration": 0, 2024-05-08T16:50:32.688 INFO:tasks.workunit.client.0.smithi172.stdout: "stat_sum": { 2024-05-08T16:50:32.688 INFO:tasks.workunit.client.0.smithi172.stdout: "num_bytes": 413696, 2024-05-08T16:50:32.688 INFO:tasks.workunit.client.0.smithi172.stdout: "num_objects": 101, 2024-05-08T16:50:32.688 INFO:tasks.workunit.client.0.smithi172.stdout: "num_object_clones": 0, 2024-05-08T16:50:32.688 INFO:tasks.workunit.client.0.smithi172.stdout: "num_object_copies": 505, 2024-05-08T16:50:32.688 INFO:tasks.workunit.client.0.smithi172.stdout: "num_objects_missing_on_primary": 1, 2024-05-08T16:50:32.688 INFO:tasks.workunit.client.0.smithi172.stdout: "num_objects_missing": 0, 2024-05-08T16:50:32.688 INFO:tasks.workunit.client.0.smithi172.stdout: "num_objects_degraded": 3, 2024-05-08T16:50:32.688 INFO:tasks.workunit.client.0.smithi172.stdout: "num_objects_misplaced": 0, 2024-05-08T16:50:32.689 INFO:tasks.workunit.client.0.smithi172.stdout: "num_objects_unfound": 1, 2024-05-08T16:50:32.689 INFO:tasks.workunit.client.0.smithi172.stdout: "num_objects_dirty": 101, 2024-05-08T16:50:32.689 INFO:tasks.workunit.client.0.smithi172.stdout: "num_whiteouts": 0, 2024-05-08T16:50:32.689 INFO:tasks.workunit.client.0.smithi172.stdout: "num_read": 0, 2024-05-08T16:50:32.689 INFO:tasks.workunit.client.0.smithi172.stdout: "num_read_kb": 0, 2024-05-08T16:50:32.689 INFO:tasks.workunit.client.0.smithi172.stdout: "num_write": 101, 2024-05-08T16:50:32.689 INFO:tasks.workunit.client.0.smithi172.stdout: "num_write_kb": 404, 2024-05-08T16:50:32.689 INFO:tasks.workunit.client.0.smithi172.stdout: "num_scrub_errors": 0, 2024-05-08T16:50:32.689 INFO:tasks.workunit.client.0.smithi172.stdout: "num_shallow_scrub_errors": 0, 2024-05-08T16:50:32.689 INFO:tasks.workunit.client.0.smithi172.stdout: "num_deep_scrub_errors": 0, 2024-05-08T16:50:32.689 INFO:tasks.workunit.client.0.smithi172.stdout: "num_objects_recovered": 99, 2024-05-08T16:50:32.689 INFO:tasks.workunit.client.0.smithi172.stdout: "num_bytes_recovered": 405504, 2024-05-08T16:50:32.689 INFO:tasks.workunit.client.0.smithi172.stdout: "num_keys_recovered": 0, 2024-05-08T16:50:32.689 INFO:tasks.workunit.client.0.smithi172.stdout: "num_objects_omap": 0, 2024-05-08T16:50:32.690 INFO:tasks.workunit.client.0.smithi172.stdout: "num_objects_hit_set_archive": 0, 2024-05-08T16:50:32.690 INFO:tasks.workunit.client.0.smithi172.stdout: "num_bytes_hit_set_archive": 0, 2024-05-08T16:50:32.690 INFO:tasks.workunit.client.0.smithi172.stdout: "num_flush": 0, 2024-05-08T16:50:32.690 INFO:tasks.workunit.client.0.smithi172.stdout: "num_flush_kb": 0, 2024-05-08T16:50:32.690 INFO:tasks.workunit.client.0.smithi172.stdout: "num_evict": 0, 2024-05-08T16:50:32.690 INFO:tasks.workunit.client.0.smithi172.stdout: "num_evict_kb": 0, 2024-05-08T16:50:32.690 INFO:tasks.workunit.client.0.smithi172.stdout: "num_promote": 0, 2024-05-08T16:50:32.690 INFO:tasks.workunit.client.0.smithi172.stdout: "num_flush_mode_high": 0, 2024-05-08T16:50:32.690 INFO:tasks.workunit.client.0.smithi172.stdout: "num_flush_mode_low": 0, 2024-05-08T16:50:32.690 INFO:tasks.workunit.client.0.smithi172.stdout: "num_evict_mode_some": 0, 2024-05-08T16:50:32.690 INFO:tasks.workunit.client.0.smithi172.stdout: "num_evict_mode_full": 0, 2024-05-08T16:50:32.690 INFO:tasks.workunit.client.0.smithi172.stdout: "num_objects_pinned": 0, 2024-05-08T16:50:32.691 INFO:tasks.workunit.client.0.smithi172.stdout: "num_legacy_snapsets": 0, 2024-05-08T16:50:32.691 INFO:tasks.workunit.client.0.smithi172.stdout: "num_large_omap_objects": 0, 2024-05-08T16:50:32.691 INFO:tasks.workunit.client.0.smithi172.stdout: "num_objects_manifest": 0, 2024-05-08T16:50:32.691 INFO:tasks.workunit.client.0.smithi172.stdout: "num_omap_bytes": 0, 2024-05-08T16:50:32.691 INFO:tasks.workunit.client.0.smithi172.stdout: "num_omap_keys": 0, 2024-05-08T16:50:32.691 INFO:tasks.workunit.client.0.smithi172.stdout: "num_objects_repaired": 0 2024-05-08T16:50:32.691 INFO:tasks.workunit.client.0.smithi172.stdout: }, 2024-05-08T16:50:32.691 INFO:tasks.workunit.client.0.smithi172.stdout: "up": [ 2024-05-08T16:50:32.691 INFO:tasks.workunit.client.0.smithi172.stdout: 3, 2024-05-08T16:50:32.691 INFO:tasks.workunit.client.0.smithi172.stdout: 1, 2024-05-08T16:50:32.691 INFO:tasks.workunit.client.0.smithi172.stdout: 4, 2024-05-08T16:50:32.691 INFO:tasks.workunit.client.0.smithi172.stdout: 0, 2024-05-08T16:50:32.691 INFO:tasks.workunit.client.0.smithi172.stdout: 2 2024-05-08T16:50:32.691 INFO:tasks.workunit.client.0.smithi172.stdout: ], 2024-05-08T16:50:32.692 INFO:tasks.workunit.client.0.smithi172.stdout: "acting": [ 2024-05-08T16:50:32.692 INFO:tasks.workunit.client.0.smithi172.stdout: 3, 2024-05-08T16:50:32.692 INFO:tasks.workunit.client.0.smithi172.stdout: 1, 2024-05-08T16:50:32.692 INFO:tasks.workunit.client.0.smithi172.stdout: 4, 2024-05-08T16:50:32.692 INFO:tasks.workunit.client.0.smithi172.stdout: 0, 2024-05-08T16:50:32.692 INFO:tasks.workunit.client.0.smithi172.stdout: 2 2024-05-08T16:50:32.692 INFO:tasks.workunit.client.0.smithi172.stdout: ], 2024-05-08T16:50:32.692 INFO:tasks.workunit.client.0.smithi172.stdout: "avail_no_missing": [ 2024-05-08T16:50:32.692 INFO:tasks.workunit.client.0.smithi172.stdout: "0(3)", 2024-05-08T16:50:32.692 INFO:tasks.workunit.client.0.smithi172.stdout: "4(2)" 2024-05-08T16:50:32.692 INFO:tasks.workunit.client.0.smithi172.stdout: ], 2024-05-08T16:50:32.692 INFO:tasks.workunit.client.0.smithi172.stdout: "object_location_counts": [ 2024-05-08T16:50:32.692 INFO:tasks.workunit.client.0.smithi172.stdout: { 2024-05-08T16:50:32.693 INFO:tasks.workunit.client.0.smithi172.stdout: "shards": "0(3),1(1),2(4),3(0),4(2)", 2024-05-08T16:50:32.693 INFO:tasks.workunit.client.0.smithi172.stdout: "objects": 100 2024-05-08T16:50:32.693 INFO:tasks.workunit.client.0.smithi172.stdout: }, 2024-05-08T16:50:32.693 INFO:tasks.workunit.client.0.smithi172.stdout: { 2024-05-08T16:50:32.693 INFO:tasks.workunit.client.0.smithi172.stdout: "shards": "0(3),4(2)", 2024-05-08T16:50:32.693 INFO:tasks.workunit.client.0.smithi172.stdout: "objects": 1 2024-05-08T16:50:32.693 INFO:tasks.workunit.client.0.smithi172.stdout: } 2024-05-08T16:50:32.693 INFO:tasks.workunit.client.0.smithi172.stdout: ], 2024-05-08T16:50:32.693 INFO:tasks.workunit.client.0.smithi172.stdout: "blocked_by": [], 2024-05-08T16:50:32.693 INFO:tasks.workunit.client.0.smithi172.stdout: "up_primary": 3, 2024-05-08T16:50:32.693 INFO:tasks.workunit.client.0.smithi172.stdout: "acting_primary": 3, 2024-05-08T16:50:32.693 INFO:tasks.workunit.client.0.smithi172.stdout: "purged_snaps": [] 2024-05-08T16:50:32.693 INFO:tasks.workunit.client.0.smithi172.stdout: }, 2024-05-08T16:50:32.694 INFO:tasks.workunit.client.0.smithi172.stdout: "empty": 0, 2024-05-08T16:50:32.694 INFO:tasks.workunit.client.0.smithi172.stdout: "dne": 0, 2024-05-08T16:50:32.694 INFO:tasks.workunit.client.0.smithi172.stdout: "incomplete": 0, 2024-05-08T16:50:32.694 INFO:tasks.workunit.client.0.smithi172.stdout: "last_epoch_started": 58, 2024-05-08T16:50:32.694 INFO:tasks.workunit.client.0.smithi172.stdout: "hit_set_history": { 2024-05-08T16:50:32.694 INFO:tasks.workunit.client.0.smithi172.stdout: "current_last_update": "0'0", 2024-05-08T16:50:32.694 INFO:tasks.workunit.client.0.smithi172.stdout: "history": [] 2024-05-08T16:50:32.694 INFO:tasks.workunit.client.0.smithi172.stdout: } 2024-05-08T16:50:32.694 INFO:tasks.workunit.client.0.smithi172.stdout: }, 2024-05-08T16:50:32.694 INFO:tasks.workunit.client.0.smithi172.stdout: "peer_info": [ 2024-05-08T16:50:32.694 INFO:tasks.workunit.client.0.smithi172.stdout: { 2024-05-08T16:50:32.694 INFO:tasks.workunit.client.0.smithi172.stdout: "peer": "0(3)", 2024-05-08T16:50:32.694 INFO:tasks.workunit.client.0.smithi172.stdout: "pgid": "2.0s3", 2024-05-08T16:50:32.694 INFO:tasks.workunit.client.0.smithi172.stdout: "shared": "3", 2024-05-08T16:50:32.694 INFO:tasks.workunit.client.0.smithi172.stdout: "last_update": "51'101", 2024-05-08T16:50:32.695 INFO:tasks.workunit.client.0.smithi172.stdout: "last_complete": "51'101", 2024-05-08T16:50:32.695 INFO:tasks.workunit.client.0.smithi172.stdout: "log_tail": "0'0", 2024-05-08T16:50:32.695 INFO:tasks.workunit.client.0.smithi172.stdout: "last_user_version": 101, 2024-05-08T16:50:32.695 INFO:tasks.workunit.client.0.smithi172.stdout: "last_backfill": "MAX", 2024-05-08T16:50:32.695 INFO:tasks.workunit.client.0.smithi172.stdout: "purged_snaps": [], 2024-05-08T16:50:32.695 INFO:tasks.workunit.client.0.smithi172.stdout: "history": { 2024-05-08T16:50:32.695 INFO:tasks.workunit.client.0.smithi172.stdout: "epoch_created": 43, 2024-05-08T16:50:32.695 INFO:tasks.workunit.client.0.smithi172.stdout: "epoch_pool_created": 43, 2024-05-08T16:50:32.695 INFO:tasks.workunit.client.0.smithi172.stdout: "last_epoch_started": 58, 2024-05-08T16:50:32.695 INFO:tasks.workunit.client.0.smithi172.stdout: "last_interval_started": 55, 2024-05-08T16:50:32.695 INFO:tasks.workunit.client.0.smithi172.stdout: "last_epoch_clean": 44, 2024-05-08T16:50:32.696 INFO:tasks.workunit.client.0.smithi172.stdout: "last_interval_clean": 43, 2024-05-08T16:50:32.696 INFO:tasks.workunit.client.0.smithi172.stdout: "last_epoch_split": 0, 2024-05-08T16:50:32.696 INFO:tasks.workunit.client.0.smithi172.stdout: "last_epoch_marked_full": 0, 2024-05-08T16:50:32.696 INFO:tasks.workunit.client.0.smithi172.stdout: "same_up_since": 55, 2024-05-08T16:50:32.696 INFO:tasks.workunit.client.0.smithi172.stdout: "same_interval_since": 55, 2024-05-08T16:50:32.696 INFO:tasks.workunit.client.0.smithi172.stdout: "same_primary_since": 43, 2024-05-08T16:50:32.696 INFO:tasks.workunit.client.0.smithi172.stdout: "last_scrub": "0'0", 2024-05-08T16:50:32.696 INFO:tasks.workunit.client.0.smithi172.stdout: "last_scrub_stamp": "2024-05-08T16:49:25.965607+0000", 2024-05-08T16:50:32.696 INFO:tasks.workunit.client.0.smithi172.stdout: "last_deep_scrub": "0'0", 2024-05-08T16:50:32.696 INFO:tasks.workunit.client.0.smithi172.stdout: "last_deep_scrub_stamp": "2024-05-08T16:49:25.965607+0000", 2024-05-08T16:50:32.696 INFO:tasks.workunit.client.0.smithi172.stdout: "last_clean_scrub_stamp": "2024-05-08T16:49:25.965607+0000", 2024-05-08T16:50:32.696 INFO:tasks.workunit.client.0.smithi172.stdout: "prior_readable_until_ub": 0 2024-05-08T16:50:32.697 INFO:tasks.workunit.client.0.smithi172.stdout: }, 2024-05-08T16:50:32.697 INFO:tasks.workunit.client.0.smithi172.stdout: "stats": { 2024-05-08T16:50:32.697 INFO:tasks.workunit.client.0.smithi172.stdout: "version": "51'101", 2024-05-08T16:50:32.697 INFO:tasks.workunit.client.0.smithi172.stdout: "reported_seq": 133, 2024-05-08T16:50:32.697 INFO:tasks.workunit.client.0.smithi172.stdout: "reported_epoch": 51, 2024-05-08T16:50:32.697 INFO:tasks.workunit.client.0.smithi172.stdout: "state": "active+undersized+degraded", 2024-05-08T16:50:32.697 INFO:tasks.workunit.client.0.smithi172.stdout: "last_fresh": "2024-05-08T16:50:08.881951+0000", 2024-05-08T16:50:32.697 INFO:tasks.workunit.client.0.smithi172.stdout: "last_change": "2024-05-08T16:49:35.993333+0000", 2024-05-08T16:50:32.697 INFO:tasks.workunit.client.0.smithi172.stdout: "last_active": "2024-05-08T16:50:08.881951+0000", 2024-05-08T16:50:32.697 INFO:tasks.workunit.client.0.smithi172.stdout: "last_peered": "2024-05-08T16:50:08.881951+0000", 2024-05-08T16:50:32.697 INFO:tasks.workunit.client.0.smithi172.stdout: "last_clean": "2024-05-08T16:49:33.679662+0000", 2024-05-08T16:50:32.697 INFO:tasks.workunit.client.0.smithi172.stdout: "last_became_active": "2024-05-08T16:49:35.993333+0000", 2024-05-08T16:50:32.697 INFO:tasks.workunit.client.0.smithi172.stdout: "last_became_peered": "2024-05-08T16:49:35.993333+0000", 2024-05-08T16:50:32.697 INFO:tasks.workunit.client.0.smithi172.stdout: "last_unstale": "2024-05-08T16:50:08.881951+0000", 2024-05-08T16:50:32.697 INFO:tasks.workunit.client.0.smithi172.stdout: "last_undegraded": "2024-05-08T16:49:35.587590+0000", 2024-05-08T16:50:32.698 INFO:tasks.workunit.client.0.smithi172.stdout: "last_fullsized": "2024-05-08T16:49:35.587392+0000", 2024-05-08T16:50:32.698 INFO:tasks.workunit.client.0.smithi172.stdout: "mapping_epoch": 55, 2024-05-08T16:50:32.698 INFO:tasks.workunit.client.0.smithi172.stdout: "log_start": "0'0", 2024-05-08T16:50:32.698 INFO:tasks.workunit.client.0.smithi172.stdout: "ondisk_log_start": "0'0", 2024-05-08T16:50:32.698 INFO:tasks.workunit.client.0.smithi172.stdout: "created": 43, 2024-05-08T16:50:32.698 INFO:tasks.workunit.client.0.smithi172.stdout: "last_epoch_clean": 44, 2024-05-08T16:50:32.698 INFO:tasks.workunit.client.0.smithi172.stdout: "parent": "0.0", 2024-05-08T16:50:32.698 INFO:tasks.workunit.client.0.smithi172.stdout: "parent_split_bits": 0, 2024-05-08T16:50:32.698 INFO:tasks.workunit.client.0.smithi172.stdout: "last_scrub": "0'0", 2024-05-08T16:50:32.698 INFO:tasks.workunit.client.0.smithi172.stdout: "last_scrub_stamp": "2024-05-08T16:49:25.965607+0000", 2024-05-08T16:50:32.698 INFO:tasks.workunit.client.0.smithi172.stdout: "last_deep_scrub": "0'0", 2024-05-08T16:50:32.698 INFO:tasks.workunit.client.0.smithi172.stdout: "last_deep_scrub_stamp": "2024-05-08T16:49:25.965607+0000", 2024-05-08T16:50:32.698 INFO:tasks.workunit.client.0.smithi172.stdout: "last_clean_scrub_stamp": "2024-05-08T16:49:25.965607+0000", 2024-05-08T16:50:32.698 INFO:tasks.workunit.client.0.smithi172.stdout: "objects_scrubbed": 0, 2024-05-08T16:50:32.699 INFO:tasks.workunit.client.0.smithi172.stdout: "log_size": 101, 2024-05-08T16:50:32.699 INFO:tasks.workunit.client.0.smithi172.stdout: "log_dups_size": 0, 2024-05-08T16:50:32.699 INFO:tasks.workunit.client.0.smithi172.stdout: "ondisk_log_size": 101, 2024-05-08T16:50:32.699 INFO:tasks.workunit.client.0.smithi172.stdout: "stats_invalid": false, 2024-05-08T16:50:32.699 INFO:tasks.workunit.client.0.smithi172.stdout: "dirty_stats_invalid": false, 2024-05-08T16:50:32.699 INFO:tasks.workunit.client.0.smithi172.stdout: "omap_stats_invalid": false, 2024-05-08T16:50:32.699 INFO:tasks.workunit.client.0.smithi172.stdout: "hitset_stats_invalid": false, 2024-05-08T16:50:32.699 INFO:tasks.workunit.client.0.smithi172.stdout: "hitset_bytes_stats_invalid": false, 2024-05-08T16:50:32.699 INFO:tasks.workunit.client.0.smithi172.stdout: "pin_stats_invalid": false, 2024-05-08T16:50:32.699 INFO:tasks.workunit.client.0.smithi172.stdout: "manifest_stats_invalid": false, 2024-05-08T16:50:32.699 INFO:tasks.workunit.client.0.smithi172.stdout: "snaptrimq_len": 0, 2024-05-08T16:50:32.699 INFO:tasks.workunit.client.0.smithi172.stdout: "last_scrub_duration": 0, 2024-05-08T16:50:32.699 INFO:tasks.workunit.client.0.smithi172.stdout: "scrub_schedule": "periodic scrub scheduled @ 2024-05-10T04:08:32.277486+0000", 2024-05-08T16:50:32.699 INFO:tasks.workunit.client.0.smithi172.stdout: "scrub_duration": 0, 2024-05-08T16:50:32.701 INFO:tasks.workunit.client.0.smithi172.stdout: "objects_trimmed": 0, 2024-05-08T16:50:32.701 INFO:tasks.workunit.client.0.smithi172.stdout: "snaptrim_duration": 0, 2024-05-08T16:50:32.701 INFO:tasks.workunit.client.0.smithi172.stdout: "stat_sum": { 2024-05-08T16:50:32.701 INFO:tasks.workunit.client.0.smithi172.stdout: "num_bytes": 413696, 2024-05-08T16:50:32.701 INFO:tasks.workunit.client.0.smithi172.stdout: "num_objects": 101, 2024-05-08T16:50:32.701 INFO:tasks.workunit.client.0.smithi172.stdout: "num_object_clones": 0, 2024-05-08T16:50:32.701 INFO:tasks.workunit.client.0.smithi172.stdout: "num_object_copies": 505, 2024-05-08T16:50:32.701 INFO:tasks.workunit.client.0.smithi172.stdout: "num_objects_missing_on_primary": 0, 2024-05-08T16:50:32.701 INFO:tasks.workunit.client.0.smithi172.stdout: "num_objects_missing": 0, 2024-05-08T16:50:32.701 INFO:tasks.workunit.client.0.smithi172.stdout: "num_objects_degraded": 101, 2024-05-08T16:50:32.701 INFO:tasks.workunit.client.0.smithi172.stdout: "num_objects_misplaced": 0, 2024-05-08T16:50:32.701 INFO:tasks.workunit.client.0.smithi172.stdout: "num_objects_unfound": 0, 2024-05-08T16:50:32.702 INFO:tasks.workunit.client.0.smithi172.stdout: "num_objects_dirty": 101, 2024-05-08T16:50:32.702 INFO:tasks.workunit.client.0.smithi172.stdout: "num_whiteouts": 0, 2024-05-08T16:50:32.702 INFO:tasks.workunit.client.0.smithi172.stdout: "num_read": 0, 2024-05-08T16:50:32.702 INFO:tasks.workunit.client.0.smithi172.stdout: "num_read_kb": 0, 2024-05-08T16:50:32.702 INFO:tasks.workunit.client.0.smithi172.stdout: "num_write": 101, 2024-05-08T16:50:32.702 INFO:tasks.workunit.client.0.smithi172.stdout: "num_write_kb": 404, 2024-05-08T16:50:32.702 INFO:tasks.workunit.client.0.smithi172.stdout: "num_scrub_errors": 0, 2024-05-08T16:50:32.702 INFO:tasks.workunit.client.0.smithi172.stdout: "num_shallow_scrub_errors": 0, 2024-05-08T16:50:32.702 INFO:tasks.workunit.client.0.smithi172.stdout: "num_deep_scrub_errors": 0, 2024-05-08T16:50:32.702 INFO:tasks.workunit.client.0.smithi172.stdout: "num_objects_recovered": 0, 2024-05-08T16:50:32.702 INFO:tasks.workunit.client.0.smithi172.stdout: "num_bytes_recovered": 0, 2024-05-08T16:50:32.702 INFO:tasks.workunit.client.0.smithi172.stdout: "num_keys_recovered": 0, 2024-05-08T16:50:32.702 INFO:tasks.workunit.client.0.smithi172.stdout: "num_objects_omap": 0, 2024-05-08T16:50:32.702 INFO:tasks.workunit.client.0.smithi172.stdout: "num_objects_hit_set_archive": 0, 2024-05-08T16:50:32.703 INFO:tasks.workunit.client.0.smithi172.stdout: "num_bytes_hit_set_archive": 0, 2024-05-08T16:50:32.703 INFO:tasks.workunit.client.0.smithi172.stdout: "num_flush": 0, 2024-05-08T16:50:32.703 INFO:tasks.workunit.client.0.smithi172.stdout: "num_flush_kb": 0, 2024-05-08T16:50:32.703 INFO:tasks.workunit.client.0.smithi172.stdout: "num_evict": 0, 2024-05-08T16:50:32.703 INFO:tasks.workunit.client.0.smithi172.stdout: "num_evict_kb": 0, 2024-05-08T16:50:32.703 INFO:tasks.workunit.client.0.smithi172.stdout: "num_promote": 0, 2024-05-08T16:50:32.703 INFO:tasks.workunit.client.0.smithi172.stdout: "num_flush_mode_high": 0, 2024-05-08T16:50:32.703 INFO:tasks.workunit.client.0.smithi172.stdout: "num_flush_mode_low": 0, 2024-05-08T16:50:32.703 INFO:tasks.workunit.client.0.smithi172.stdout: "num_evict_mode_some": 0, 2024-05-08T16:50:32.703 INFO:tasks.workunit.client.0.smithi172.stdout: "num_evict_mode_full": 0, 2024-05-08T16:50:32.703 INFO:tasks.workunit.client.0.smithi172.stdout: "num_objects_pinned": 0, 2024-05-08T16:50:32.703 INFO:tasks.workunit.client.0.smithi172.stdout: "num_legacy_snapsets": 0, 2024-05-08T16:50:32.703 INFO:tasks.workunit.client.0.smithi172.stdout: "num_large_omap_objects": 0, 2024-05-08T16:50:32.703 INFO:tasks.workunit.client.0.smithi172.stdout: "num_objects_manifest": 0, 2024-05-08T16:50:32.703 INFO:tasks.workunit.client.0.smithi172.stdout: "num_omap_bytes": 0, 2024-05-08T16:50:32.703 INFO:tasks.workunit.client.0.smithi172.stdout: "num_omap_keys": 0, 2024-05-08T16:50:32.704 INFO:tasks.workunit.client.0.smithi172.stdout: "num_objects_repaired": 0 2024-05-08T16:50:32.704 INFO:tasks.workunit.client.0.smithi172.stdout: }, 2024-05-08T16:50:32.704 INFO:tasks.workunit.client.0.smithi172.stdout: "up": [ 2024-05-08T16:50:32.704 INFO:tasks.workunit.client.0.smithi172.stdout: 3, 2024-05-08T16:50:32.704 INFO:tasks.workunit.client.0.smithi172.stdout: 1, 2024-05-08T16:50:32.704 INFO:tasks.workunit.client.0.smithi172.stdout: 4, 2024-05-08T16:50:32.704 INFO:tasks.workunit.client.0.smithi172.stdout: 0, 2024-05-08T16:50:32.704 INFO:tasks.workunit.client.0.smithi172.stdout: 2 2024-05-08T16:50:32.704 INFO:tasks.workunit.client.0.smithi172.stdout: ], 2024-05-08T16:50:32.704 INFO:tasks.workunit.client.0.smithi172.stdout: "acting": [ 2024-05-08T16:50:32.704 INFO:tasks.workunit.client.0.smithi172.stdout: 3, 2024-05-08T16:50:32.704 INFO:tasks.workunit.client.0.smithi172.stdout: 1, 2024-05-08T16:50:32.704 INFO:tasks.workunit.client.0.smithi172.stdout: 4, 2024-05-08T16:50:32.704 INFO:tasks.workunit.client.0.smithi172.stdout: 0, 2024-05-08T16:50:32.704 INFO:tasks.workunit.client.0.smithi172.stdout: 2 2024-05-08T16:50:32.704 INFO:tasks.workunit.client.0.smithi172.stdout: ], 2024-05-08T16:50:32.705 INFO:tasks.workunit.client.0.smithi172.stdout: "avail_no_missing": [ 2024-05-08T16:50:32.705 INFO:tasks.workunit.client.0.smithi172.stdout: "3(0)", 2024-05-08T16:50:32.705 INFO:tasks.workunit.client.0.smithi172.stdout: "0(3)", 2024-05-08T16:50:32.705 INFO:tasks.workunit.client.0.smithi172.stdout: "1(1)", 2024-05-08T16:50:32.705 INFO:tasks.workunit.client.0.smithi172.stdout: "4(2)" 2024-05-08T16:50:32.705 INFO:tasks.workunit.client.0.smithi172.stdout: ], 2024-05-08T16:50:32.705 INFO:tasks.workunit.client.0.smithi172.stdout: "object_location_counts": [ 2024-05-08T16:50:32.705 INFO:tasks.workunit.client.0.smithi172.stdout: { 2024-05-08T16:50:32.705 INFO:tasks.workunit.client.0.smithi172.stdout: "shards": "0(3),1(1),3(0),4(2)", 2024-05-08T16:50:32.705 INFO:tasks.workunit.client.0.smithi172.stdout: "objects": 101 2024-05-08T16:50:32.705 INFO:tasks.workunit.client.0.smithi172.stdout: } 2024-05-08T16:50:32.705 INFO:tasks.workunit.client.0.smithi172.stdout: ], 2024-05-08T16:50:32.705 INFO:tasks.workunit.client.0.smithi172.stdout: "blocked_by": [], 2024-05-08T16:50:32.705 INFO:tasks.workunit.client.0.smithi172.stdout: "up_primary": 3, 2024-05-08T16:50:32.706 INFO:tasks.workunit.client.0.smithi172.stdout: "acting_primary": 3, 2024-05-08T16:50:32.706 INFO:tasks.workunit.client.0.smithi172.stdout: "purged_snaps": [] 2024-05-08T16:50:32.706 INFO:tasks.workunit.client.0.smithi172.stdout: }, 2024-05-08T16:50:32.706 INFO:tasks.workunit.client.0.smithi172.stdout: "empty": 0, 2024-05-08T16:50:32.706 INFO:tasks.workunit.client.0.smithi172.stdout: "dne": 0, 2024-05-08T16:50:32.706 INFO:tasks.workunit.client.0.smithi172.stdout: "incomplete": 0, 2024-05-08T16:50:32.706 INFO:tasks.workunit.client.0.smithi172.stdout: "last_epoch_started": 58, 2024-05-08T16:50:32.706 INFO:tasks.workunit.client.0.smithi172.stdout: "hit_set_history": { 2024-05-08T16:50:32.706 INFO:tasks.workunit.client.0.smithi172.stdout: "current_last_update": "0'0", 2024-05-08T16:50:32.706 INFO:tasks.workunit.client.0.smithi172.stdout: "history": [] 2024-05-08T16:50:32.706 INFO:tasks.workunit.client.0.smithi172.stdout: } 2024-05-08T16:50:32.706 INFO:tasks.workunit.client.0.smithi172.stdout: }, 2024-05-08T16:50:32.706 INFO:tasks.workunit.client.0.smithi172.stdout: { 2024-05-08T16:50:32.706 INFO:tasks.workunit.client.0.smithi172.stdout: "peer": "1(1)", 2024-05-08T16:50:32.706 INFO:tasks.workunit.client.0.smithi172.stdout: "pgid": "2.0s1", 2024-05-08T16:50:32.707 INFO:tasks.workunit.client.0.smithi172.stdout: "shared": "1", 2024-05-08T16:50:32.707 INFO:tasks.workunit.client.0.smithi172.stdout: "last_update": "51'101", 2024-05-08T16:50:32.707 INFO:tasks.workunit.client.0.smithi172.stdout: "last_complete": "51'101", 2024-05-08T16:50:32.707 INFO:tasks.workunit.client.0.smithi172.stdout: "log_tail": "0'0", 2024-05-08T16:50:32.707 INFO:tasks.workunit.client.0.smithi172.stdout: "last_user_version": 101, 2024-05-08T16:50:32.707 INFO:tasks.workunit.client.0.smithi172.stdout: "last_backfill": "MAX", 2024-05-08T16:50:32.707 INFO:tasks.workunit.client.0.smithi172.stdout: "purged_snaps": [], 2024-05-08T16:50:32.707 INFO:tasks.workunit.client.0.smithi172.stdout: "history": { 2024-05-08T16:50:32.707 INFO:tasks.workunit.client.0.smithi172.stdout: "epoch_created": 43, 2024-05-08T16:50:32.707 INFO:tasks.workunit.client.0.smithi172.stdout: "epoch_pool_created": 43, 2024-05-08T16:50:32.707 INFO:tasks.workunit.client.0.smithi172.stdout: "last_epoch_started": 58, 2024-05-08T16:50:32.707 INFO:tasks.workunit.client.0.smithi172.stdout: "last_interval_started": 55, 2024-05-08T16:50:32.707 INFO:tasks.workunit.client.0.smithi172.stdout: "last_epoch_clean": 44, 2024-05-08T16:50:32.707 INFO:tasks.workunit.client.0.smithi172.stdout: "last_interval_clean": 43, 2024-05-08T16:50:32.707 INFO:tasks.workunit.client.0.smithi172.stdout: "last_epoch_split": 0, 2024-05-08T16:50:32.708 INFO:tasks.workunit.client.0.smithi172.stdout: "last_epoch_marked_full": 0, 2024-05-08T16:50:32.708 INFO:tasks.workunit.client.0.smithi172.stdout: "same_up_since": 55, 2024-05-08T16:50:32.708 INFO:tasks.workunit.client.0.smithi172.stdout: "same_interval_since": 55, 2024-05-08T16:50:32.708 INFO:tasks.workunit.client.0.smithi172.stdout: "same_primary_since": 43, 2024-05-08T16:50:32.708 INFO:tasks.workunit.client.0.smithi172.stdout: "last_scrub": "0'0", 2024-05-08T16:50:32.708 INFO:tasks.workunit.client.0.smithi172.stdout: "last_scrub_stamp": "2024-05-08T16:49:25.965607+0000", 2024-05-08T16:50:32.708 INFO:tasks.workunit.client.0.smithi172.stdout: "last_deep_scrub": "0'0", 2024-05-08T16:50:32.708 INFO:tasks.workunit.client.0.smithi172.stdout: "last_deep_scrub_stamp": "2024-05-08T16:49:25.965607+0000", 2024-05-08T16:50:32.708 INFO:tasks.workunit.client.0.smithi172.stdout: "last_clean_scrub_stamp": "2024-05-08T16:49:25.965607+0000", 2024-05-08T16:50:32.708 INFO:tasks.workunit.client.0.smithi172.stdout: "prior_readable_until_ub": 0 2024-05-08T16:50:32.708 INFO:tasks.workunit.client.0.smithi172.stdout: }, 2024-05-08T16:50:32.708 INFO:tasks.workunit.client.0.smithi172.stdout: "stats": { 2024-05-08T16:50:32.708 INFO:tasks.workunit.client.0.smithi172.stdout: "version": "51'101", 2024-05-08T16:50:32.708 INFO:tasks.workunit.client.0.smithi172.stdout: "reported_seq": 133, 2024-05-08T16:50:32.708 INFO:tasks.workunit.client.0.smithi172.stdout: "reported_epoch": 51, 2024-05-08T16:50:32.709 INFO:tasks.workunit.client.0.smithi172.stdout: "state": "active+undersized+degraded", 2024-05-08T16:50:32.709 INFO:tasks.workunit.client.0.smithi172.stdout: "last_fresh": "2024-05-08T16:50:08.881951+0000", 2024-05-08T16:50:32.709 INFO:tasks.workunit.client.0.smithi172.stdout: "last_change": "2024-05-08T16:49:35.993333+0000", 2024-05-08T16:50:32.709 INFO:tasks.workunit.client.0.smithi172.stdout: "last_active": "2024-05-08T16:50:08.881951+0000", 2024-05-08T16:50:32.709 INFO:tasks.workunit.client.0.smithi172.stdout: "last_peered": "2024-05-08T16:50:08.881951+0000", 2024-05-08T16:50:32.709 INFO:tasks.workunit.client.0.smithi172.stdout: "last_clean": "2024-05-08T16:49:33.679662+0000", 2024-05-08T16:50:32.709 INFO:tasks.workunit.client.0.smithi172.stdout: "last_became_active": "2024-05-08T16:49:35.993333+0000", 2024-05-08T16:50:32.709 INFO:tasks.workunit.client.0.smithi172.stdout: "last_became_peered": "2024-05-08T16:49:35.993333+0000", 2024-05-08T16:50:32.709 INFO:tasks.workunit.client.0.smithi172.stdout: "last_unstale": "2024-05-08T16:50:08.881951+0000", 2024-05-08T16:50:32.709 INFO:tasks.workunit.client.0.smithi172.stdout: "last_undegraded": "2024-05-08T16:49:35.587590+0000", 2024-05-08T16:50:32.709 INFO:tasks.workunit.client.0.smithi172.stdout: "last_fullsized": "2024-05-08T16:49:35.587392+0000", 2024-05-08T16:50:32.709 INFO:tasks.workunit.client.0.smithi172.stdout: "mapping_epoch": 55, 2024-05-08T16:50:32.709 INFO:tasks.workunit.client.0.smithi172.stdout: "log_start": "0'0", 2024-05-08T16:50:32.710 INFO:tasks.workunit.client.0.smithi172.stdout: "ondisk_log_start": "0'0", 2024-05-08T16:50:32.710 INFO:tasks.workunit.client.0.smithi172.stdout: "created": 43, 2024-05-08T16:50:32.710 INFO:tasks.workunit.client.0.smithi172.stdout: "last_epoch_clean": 44, 2024-05-08T16:50:32.710 INFO:tasks.workunit.client.0.smithi172.stdout: "parent": "0.0", 2024-05-08T16:50:32.710 INFO:tasks.workunit.client.0.smithi172.stdout: "parent_split_bits": 0, 2024-05-08T16:50:32.710 INFO:tasks.workunit.client.0.smithi172.stdout: "last_scrub": "0'0", 2024-05-08T16:50:32.710 INFO:tasks.workunit.client.0.smithi172.stdout: "last_scrub_stamp": "2024-05-08T16:49:25.965607+0000", 2024-05-08T16:50:32.710 INFO:tasks.workunit.client.0.smithi172.stdout: "last_deep_scrub": "0'0", 2024-05-08T16:50:32.710 INFO:tasks.workunit.client.0.smithi172.stdout: "last_deep_scrub_stamp": "2024-05-08T16:49:25.965607+0000", 2024-05-08T16:50:32.710 INFO:tasks.workunit.client.0.smithi172.stdout: "last_clean_scrub_stamp": "2024-05-08T16:49:25.965607+0000", 2024-05-08T16:50:32.710 INFO:tasks.workunit.client.0.smithi172.stdout: "objects_scrubbed": 0, 2024-05-08T16:50:32.710 INFO:tasks.workunit.client.0.smithi172.stdout: "log_size": 101, 2024-05-08T16:50:32.710 INFO:tasks.workunit.client.0.smithi172.stdout: "log_dups_size": 0, 2024-05-08T16:50:32.710 INFO:tasks.workunit.client.0.smithi172.stdout: "ondisk_log_size": 101, 2024-05-08T16:50:32.711 INFO:tasks.workunit.client.0.smithi172.stdout: "stats_invalid": false, 2024-05-08T16:50:32.711 INFO:tasks.workunit.client.0.smithi172.stdout: "dirty_stats_invalid": false, 2024-05-08T16:50:32.711 INFO:tasks.workunit.client.0.smithi172.stdout: "omap_stats_invalid": false, 2024-05-08T16:50:32.711 INFO:tasks.workunit.client.0.smithi172.stdout: "hitset_stats_invalid": false, 2024-05-08T16:50:32.711 INFO:tasks.workunit.client.0.smithi172.stdout: "hitset_bytes_stats_invalid": false, 2024-05-08T16:50:32.711 INFO:tasks.workunit.client.0.smithi172.stdout: "pin_stats_invalid": false, 2024-05-08T16:50:32.711 INFO:tasks.workunit.client.0.smithi172.stdout: "manifest_stats_invalid": false, 2024-05-08T16:50:32.711 INFO:tasks.workunit.client.0.smithi172.stdout: "snaptrimq_len": 0, 2024-05-08T16:50:32.711 INFO:tasks.workunit.client.0.smithi172.stdout: "last_scrub_duration": 0, 2024-05-08T16:50:32.711 INFO:tasks.workunit.client.0.smithi172.stdout: "scrub_schedule": "periodic scrub scheduled @ 2024-05-10T04:08:32.277486+0000", 2024-05-08T16:50:32.711 INFO:tasks.workunit.client.0.smithi172.stdout: "scrub_duration": 0, 2024-05-08T16:50:32.711 INFO:tasks.workunit.client.0.smithi172.stdout: "objects_trimmed": 0, 2024-05-08T16:50:32.711 INFO:tasks.workunit.client.0.smithi172.stdout: "snaptrim_duration": 0, 2024-05-08T16:50:32.711 INFO:tasks.workunit.client.0.smithi172.stdout: "stat_sum": { 2024-05-08T16:50:32.711 INFO:tasks.workunit.client.0.smithi172.stdout: "num_bytes": 413696, 2024-05-08T16:50:32.712 INFO:tasks.workunit.client.0.smithi172.stdout: "num_objects": 101, 2024-05-08T16:50:32.712 INFO:tasks.workunit.client.0.smithi172.stdout: "num_object_clones": 0, 2024-05-08T16:50:32.712 INFO:tasks.workunit.client.0.smithi172.stdout: "num_object_copies": 505, 2024-05-08T16:50:32.712 INFO:tasks.workunit.client.0.smithi172.stdout: "num_objects_missing_on_primary": 0, 2024-05-08T16:50:32.712 INFO:tasks.workunit.client.0.smithi172.stdout: "num_objects_missing": 1, 2024-05-08T16:50:32.712 INFO:tasks.workunit.client.0.smithi172.stdout: "num_objects_degraded": 101, 2024-05-08T16:50:32.712 INFO:tasks.workunit.client.0.smithi172.stdout: "num_objects_misplaced": 0, 2024-05-08T16:50:32.712 INFO:tasks.workunit.client.0.smithi172.stdout: "num_objects_unfound": 0, 2024-05-08T16:50:32.712 INFO:tasks.workunit.client.0.smithi172.stdout: "num_objects_dirty": 101, 2024-05-08T16:50:32.712 INFO:tasks.workunit.client.0.smithi172.stdout: "num_whiteouts": 0, 2024-05-08T16:50:32.712 INFO:tasks.workunit.client.0.smithi172.stdout: "num_read": 0, 2024-05-08T16:50:32.712 INFO:tasks.workunit.client.0.smithi172.stdout: "num_read_kb": 0, 2024-05-08T16:50:32.712 INFO:tasks.workunit.client.0.smithi172.stdout: "num_write": 101, 2024-05-08T16:50:32.712 INFO:tasks.workunit.client.0.smithi172.stdout: "num_write_kb": 404, 2024-05-08T16:50:32.713 INFO:tasks.workunit.client.0.smithi172.stdout: "num_scrub_errors": 0, 2024-05-08T16:50:32.713 INFO:tasks.workunit.client.0.smithi172.stdout: "num_shallow_scrub_errors": 0, 2024-05-08T16:50:32.713 INFO:tasks.workunit.client.0.smithi172.stdout: "num_deep_scrub_errors": 0, 2024-05-08T16:50:32.713 INFO:tasks.workunit.client.0.smithi172.stdout: "num_objects_recovered": 0, 2024-05-08T16:50:32.713 INFO:tasks.workunit.client.0.smithi172.stdout: "num_bytes_recovered": 0, 2024-05-08T16:50:32.713 INFO:tasks.workunit.client.0.smithi172.stdout: "num_keys_recovered": 0, 2024-05-08T16:50:32.713 INFO:tasks.workunit.client.0.smithi172.stdout: "num_objects_omap": 0, 2024-05-08T16:50:32.713 INFO:tasks.workunit.client.0.smithi172.stdout: "num_objects_hit_set_archive": 0, 2024-05-08T16:50:32.713 INFO:tasks.workunit.client.0.smithi172.stdout: "num_bytes_hit_set_archive": 0, 2024-05-08T16:50:32.713 INFO:tasks.workunit.client.0.smithi172.stdout: "num_flush": 0, 2024-05-08T16:50:32.713 INFO:tasks.workunit.client.0.smithi172.stdout: "num_flush_kb": 0, 2024-05-08T16:50:32.713 INFO:tasks.workunit.client.0.smithi172.stdout: "num_evict": 0, 2024-05-08T16:50:32.713 INFO:tasks.workunit.client.0.smithi172.stdout: "num_evict_kb": 0, 2024-05-08T16:50:32.714 INFO:tasks.workunit.client.0.smithi172.stdout: "num_promote": 0, 2024-05-08T16:50:32.714 INFO:tasks.workunit.client.0.smithi172.stdout: "num_flush_mode_high": 0, 2024-05-08T16:50:32.714 INFO:tasks.workunit.client.0.smithi172.stdout: "num_flush_mode_low": 0, 2024-05-08T16:50:32.714 INFO:tasks.workunit.client.0.smithi172.stdout: "num_evict_mode_some": 0, 2024-05-08T16:50:32.714 INFO:tasks.workunit.client.0.smithi172.stdout: "num_evict_mode_full": 0, 2024-05-08T16:50:32.714 INFO:tasks.workunit.client.0.smithi172.stdout: "num_objects_pinned": 0, 2024-05-08T16:50:32.714 INFO:tasks.workunit.client.0.smithi172.stdout: "num_legacy_snapsets": 0, 2024-05-08T16:50:32.714 INFO:tasks.workunit.client.0.smithi172.stdout: "num_large_omap_objects": 0, 2024-05-08T16:50:32.714 INFO:tasks.workunit.client.0.smithi172.stdout: "num_objects_manifest": 0, 2024-05-08T16:50:32.714 INFO:tasks.workunit.client.0.smithi172.stdout: "num_omap_bytes": 0, 2024-05-08T16:50:32.714 INFO:tasks.workunit.client.0.smithi172.stdout: "num_omap_keys": 0, 2024-05-08T16:50:32.714 INFO:tasks.workunit.client.0.smithi172.stdout: "num_objects_repaired": 0 2024-05-08T16:50:32.714 INFO:tasks.workunit.client.0.smithi172.stdout: }, 2024-05-08T16:50:32.714 INFO:tasks.workunit.client.0.smithi172.stdout: "up": [ 2024-05-08T16:50:32.715 INFO:tasks.workunit.client.0.smithi172.stdout: 3, 2024-05-08T16:50:32.715 INFO:tasks.workunit.client.0.smithi172.stdout: 1, 2024-05-08T16:50:32.715 INFO:tasks.workunit.client.0.smithi172.stdout: 4, 2024-05-08T16:50:32.715 INFO:tasks.workunit.client.0.smithi172.stdout: 0, 2024-05-08T16:50:32.715 INFO:tasks.workunit.client.0.smithi172.stdout: 2 2024-05-08T16:50:32.715 INFO:tasks.workunit.client.0.smithi172.stdout: ], 2024-05-08T16:50:32.715 INFO:tasks.workunit.client.0.smithi172.stdout: "acting": [ 2024-05-08T16:50:32.715 INFO:tasks.workunit.client.0.smithi172.stdout: 3, 2024-05-08T16:50:32.715 INFO:tasks.workunit.client.0.smithi172.stdout: 1, 2024-05-08T16:50:32.715 INFO:tasks.workunit.client.0.smithi172.stdout: 4, 2024-05-08T16:50:32.715 INFO:tasks.workunit.client.0.smithi172.stdout: 0, 2024-05-08T16:50:32.715 INFO:tasks.workunit.client.0.smithi172.stdout: 2 2024-05-08T16:50:32.715 INFO:tasks.workunit.client.0.smithi172.stdout: ], 2024-05-08T16:50:32.715 INFO:tasks.workunit.client.0.smithi172.stdout: "avail_no_missing": [ 2024-05-08T16:50:32.715 INFO:tasks.workunit.client.0.smithi172.stdout: "3(0)", 2024-05-08T16:50:32.716 INFO:tasks.workunit.client.0.smithi172.stdout: "0(3)", 2024-05-08T16:50:32.716 INFO:tasks.workunit.client.0.smithi172.stdout: "1(1)", 2024-05-08T16:50:32.716 INFO:tasks.workunit.client.0.smithi172.stdout: "4(2)" 2024-05-08T16:50:32.716 INFO:tasks.workunit.client.0.smithi172.stdout: ], 2024-05-08T16:50:32.716 INFO:tasks.workunit.client.0.smithi172.stdout: "object_location_counts": [ 2024-05-08T16:50:32.716 INFO:tasks.workunit.client.0.smithi172.stdout: { 2024-05-08T16:50:32.716 INFO:tasks.workunit.client.0.smithi172.stdout: "shards": "0(3),1(1),3(0),4(2)", 2024-05-08T16:50:32.716 INFO:tasks.workunit.client.0.smithi172.stdout: "objects": 101 2024-05-08T16:50:32.716 INFO:tasks.workunit.client.0.smithi172.stdout: } 2024-05-08T16:50:32.716 INFO:tasks.workunit.client.0.smithi172.stdout: ], 2024-05-08T16:50:32.716 INFO:tasks.workunit.client.0.smithi172.stdout: "blocked_by": [], 2024-05-08T16:50:32.716 INFO:tasks.workunit.client.0.smithi172.stdout: "up_primary": 3, 2024-05-08T16:50:32.716 INFO:tasks.workunit.client.0.smithi172.stdout: "acting_primary": 3, 2024-05-08T16:50:32.717 INFO:tasks.workunit.client.0.smithi172.stdout: "purged_snaps": [] 2024-05-08T16:50:32.717 INFO:tasks.workunit.client.0.smithi172.stdout: }, 2024-05-08T16:50:32.717 INFO:tasks.workunit.client.0.smithi172.stdout: "empty": 0, 2024-05-08T16:50:32.717 INFO:tasks.workunit.client.0.smithi172.stdout: "dne": 0, 2024-05-08T16:50:32.717 INFO:tasks.workunit.client.0.smithi172.stdout: "incomplete": 0, 2024-05-08T16:50:32.717 INFO:tasks.workunit.client.0.smithi172.stdout: "last_epoch_started": 58, 2024-05-08T16:50:32.717 INFO:tasks.workunit.client.0.smithi172.stdout: "hit_set_history": { 2024-05-08T16:50:32.717 INFO:tasks.workunit.client.0.smithi172.stdout: "current_last_update": "0'0", 2024-05-08T16:50:32.717 INFO:tasks.workunit.client.0.smithi172.stdout: "history": [] 2024-05-08T16:50:32.717 INFO:tasks.workunit.client.0.smithi172.stdout: } 2024-05-08T16:50:32.717 INFO:tasks.workunit.client.0.smithi172.stdout: }, 2024-05-08T16:50:32.717 INFO:tasks.workunit.client.0.smithi172.stdout: { 2024-05-08T16:50:32.717 INFO:tasks.workunit.client.0.smithi172.stdout: "peer": "2(4)", 2024-05-08T16:50:32.717 INFO:tasks.workunit.client.0.smithi172.stdout: "pgid": "2.0s4", 2024-05-08T16:50:32.717 INFO:tasks.workunit.client.0.smithi172.stdout: "shared": "4", 2024-05-08T16:50:32.718 INFO:tasks.workunit.client.0.smithi172.stdout: "last_update": "51'101", 2024-05-08T16:50:32.718 INFO:tasks.workunit.client.0.smithi172.stdout: "last_complete": "45'1", 2024-05-08T16:50:32.718 INFO:tasks.workunit.client.0.smithi172.stdout: "log_tail": "0'0", 2024-05-08T16:50:32.718 INFO:tasks.workunit.client.0.smithi172.stdout: "last_user_version": 1, 2024-05-08T16:50:32.718 INFO:tasks.workunit.client.0.smithi172.stdout: "last_backfill": "MAX", 2024-05-08T16:50:32.718 INFO:tasks.workunit.client.0.smithi172.stdout: "purged_snaps": [], 2024-05-08T16:50:32.718 INFO:tasks.workunit.client.0.smithi172.stdout: "history": { 2024-05-08T16:50:32.718 INFO:tasks.workunit.client.0.smithi172.stdout: "epoch_created": 43, 2024-05-08T16:50:32.718 INFO:tasks.workunit.client.0.smithi172.stdout: "epoch_pool_created": 43, 2024-05-08T16:50:32.718 INFO:tasks.workunit.client.0.smithi172.stdout: "last_epoch_started": 58, 2024-05-08T16:50:32.718 INFO:tasks.workunit.client.0.smithi172.stdout: "last_interval_started": 55, 2024-05-08T16:50:32.718 INFO:tasks.workunit.client.0.smithi172.stdout: "last_epoch_clean": 44, 2024-05-08T16:50:32.719 INFO:tasks.workunit.client.0.smithi172.stdout: "last_interval_clean": 43, 2024-05-08T16:50:32.719 INFO:tasks.workunit.client.0.smithi172.stdout: "last_epoch_split": 0, 2024-05-08T16:50:32.719 INFO:tasks.workunit.client.0.smithi172.stdout: "last_epoch_marked_full": 0, 2024-05-08T16:50:32.719 INFO:tasks.workunit.client.0.smithi172.stdout: "same_up_since": 55, 2024-05-08T16:50:32.719 INFO:tasks.workunit.client.0.smithi172.stdout: "same_interval_since": 55, 2024-05-08T16:50:32.719 INFO:tasks.workunit.client.0.smithi172.stdout: "same_primary_since": 43, 2024-05-08T16:50:32.719 INFO:tasks.workunit.client.0.smithi172.stdout: "last_scrub": "0'0", 2024-05-08T16:50:32.719 INFO:tasks.workunit.client.0.smithi172.stdout: "last_scrub_stamp": "2024-05-08T16:49:25.965607+0000", 2024-05-08T16:50:32.719 INFO:tasks.workunit.client.0.smithi172.stdout: "last_deep_scrub": "0'0", 2024-05-08T16:50:32.719 INFO:tasks.workunit.client.0.smithi172.stdout: "last_deep_scrub_stamp": "2024-05-08T16:49:25.965607+0000", 2024-05-08T16:50:32.719 INFO:tasks.workunit.client.0.smithi172.stdout: "last_clean_scrub_stamp": "2024-05-08T16:49:25.965607+0000", 2024-05-08T16:50:32.719 INFO:tasks.workunit.client.0.smithi172.stdout: "prior_readable_until_ub": 0 2024-05-08T16:50:32.719 INFO:tasks.workunit.client.0.smithi172.stdout: }, 2024-05-08T16:50:32.719 INFO:tasks.workunit.client.0.smithi172.stdout: "stats": { 2024-05-08T16:50:32.719 INFO:tasks.workunit.client.0.smithi172.stdout: "version": "45'1", 2024-05-08T16:50:32.720 INFO:tasks.workunit.client.0.smithi172.stdout: "reported_seq": 17, 2024-05-08T16:50:32.720 INFO:tasks.workunit.client.0.smithi172.stdout: "reported_epoch": 45, 2024-05-08T16:50:32.720 INFO:tasks.workunit.client.0.smithi172.stdout: "state": "active+clean", 2024-05-08T16:50:32.720 INFO:tasks.workunit.client.0.smithi172.stdout: "last_fresh": "2024-05-08T16:49:33.679662+0000", 2024-05-08T16:50:32.720 INFO:tasks.workunit.client.0.smithi172.stdout: "last_change": "2024-05-08T16:49:28.257906+0000", 2024-05-08T16:50:32.720 INFO:tasks.workunit.client.0.smithi172.stdout: "last_active": "2024-05-08T16:49:33.679662+0000", 2024-05-08T16:50:32.720 INFO:tasks.workunit.client.0.smithi172.stdout: "last_peered": "2024-05-08T16:49:33.679662+0000", 2024-05-08T16:50:32.720 INFO:tasks.workunit.client.0.smithi172.stdout: "last_clean": "2024-05-08T16:49:33.679662+0000", 2024-05-08T16:50:32.720 INFO:tasks.workunit.client.0.smithi172.stdout: "last_became_active": "2024-05-08T16:49:28.256746+0000", 2024-05-08T16:50:32.720 INFO:tasks.workunit.client.0.smithi172.stdout: "last_became_peered": "2024-05-08T16:49:28.256746+0000", 2024-05-08T16:50:32.720 INFO:tasks.workunit.client.0.smithi172.stdout: "last_unstale": "2024-05-08T16:49:33.679662+0000", 2024-05-08T16:50:32.720 INFO:tasks.workunit.client.0.smithi172.stdout: "last_undegraded": "2024-05-08T16:49:33.679662+0000", 2024-05-08T16:50:32.720 INFO:tasks.workunit.client.0.smithi172.stdout: "last_fullsized": "2024-05-08T16:49:33.679662+0000", 2024-05-08T16:50:32.720 INFO:tasks.workunit.client.0.smithi172.stdout: "mapping_epoch": 55, 2024-05-08T16:50:32.721 INFO:tasks.workunit.client.0.smithi172.stdout: "log_start": "0'0", 2024-05-08T16:50:32.721 INFO:tasks.workunit.client.0.smithi172.stdout: "ondisk_log_start": "0'0", 2024-05-08T16:50:32.721 INFO:tasks.workunit.client.0.smithi172.stdout: "created": 43, 2024-05-08T16:50:32.721 INFO:tasks.workunit.client.0.smithi172.stdout: "last_epoch_clean": 44, 2024-05-08T16:50:32.721 INFO:tasks.workunit.client.0.smithi172.stdout: "parent": "0.0", 2024-05-08T16:50:32.721 INFO:tasks.workunit.client.0.smithi172.stdout: "parent_split_bits": 0, 2024-05-08T16:50:32.721 INFO:tasks.workunit.client.0.smithi172.stdout: "last_scrub": "0'0", 2024-05-08T16:50:32.721 INFO:tasks.workunit.client.0.smithi172.stdout: "last_scrub_stamp": "2024-05-08T16:49:25.965607+0000", 2024-05-08T16:50:32.721 INFO:tasks.workunit.client.0.smithi172.stdout: "last_deep_scrub": "0'0", 2024-05-08T16:50:32.721 INFO:tasks.workunit.client.0.smithi172.stdout: "last_deep_scrub_stamp": "2024-05-08T16:49:25.965607+0000", 2024-05-08T16:50:32.721 INFO:tasks.workunit.client.0.smithi172.stdout: "last_clean_scrub_stamp": "2024-05-08T16:49:25.965607+0000", 2024-05-08T16:50:32.721 INFO:tasks.workunit.client.0.smithi172.stdout: "objects_scrubbed": 0, 2024-05-08T16:50:32.721 INFO:tasks.workunit.client.0.smithi172.stdout: "log_size": 1, 2024-05-08T16:50:32.721 INFO:tasks.workunit.client.0.smithi172.stdout: "log_dups_size": 0, 2024-05-08T16:50:32.721 INFO:tasks.workunit.client.0.smithi172.stdout: "ondisk_log_size": 1, 2024-05-08T16:50:32.721 INFO:tasks.workunit.client.0.smithi172.stdout: "stats_invalid": false, 2024-05-08T16:50:32.722 INFO:tasks.workunit.client.0.smithi172.stdout: "dirty_stats_invalid": false, 2024-05-08T16:50:32.722 INFO:tasks.workunit.client.0.smithi172.stdout: "omap_stats_invalid": false, 2024-05-08T16:50:32.722 INFO:tasks.workunit.client.0.smithi172.stdout: "hitset_stats_invalid": false, 2024-05-08T16:50:32.722 INFO:tasks.workunit.client.0.smithi172.stdout: "hitset_bytes_stats_invalid": false, 2024-05-08T16:50:32.722 INFO:tasks.workunit.client.0.smithi172.stdout: "pin_stats_invalid": false, 2024-05-08T16:50:32.722 INFO:tasks.workunit.client.0.smithi172.stdout: "manifest_stats_invalid": false, 2024-05-08T16:50:32.722 INFO:tasks.workunit.client.0.smithi172.stdout: "snaptrimq_len": 0, 2024-05-08T16:50:32.722 INFO:tasks.workunit.client.0.smithi172.stdout: "last_scrub_duration": 0, 2024-05-08T16:50:32.722 INFO:tasks.workunit.client.0.smithi172.stdout: "scrub_schedule": "periodic scrub scheduled @ 2024-05-10T03:56:04.764441+0000", 2024-05-08T16:50:32.722 INFO:tasks.workunit.client.0.smithi172.stdout: "scrub_duration": 0, 2024-05-08T16:50:32.722 INFO:tasks.workunit.client.0.smithi172.stdout: "objects_trimmed": 0, 2024-05-08T16:50:32.722 INFO:tasks.workunit.client.0.smithi172.stdout: "snaptrim_duration": 0, 2024-05-08T16:50:32.722 INFO:tasks.workunit.client.0.smithi172.stdout: "stat_sum": { 2024-05-08T16:50:32.722 INFO:tasks.workunit.client.0.smithi172.stdout: "num_bytes": 4096, 2024-05-08T16:50:32.722 INFO:tasks.workunit.client.0.smithi172.stdout: "num_objects": 1, 2024-05-08T16:50:32.723 INFO:tasks.workunit.client.0.smithi172.stdout: "num_object_clones": 0, 2024-05-08T16:50:32.723 INFO:tasks.workunit.client.0.smithi172.stdout: "num_object_copies": 5, 2024-05-08T16:50:32.723 INFO:tasks.workunit.client.0.smithi172.stdout: "num_objects_missing_on_primary": 0, 2024-05-08T16:50:32.723 INFO:tasks.workunit.client.0.smithi172.stdout: "num_objects_missing": 1, 2024-05-08T16:50:32.723 INFO:tasks.workunit.client.0.smithi172.stdout: "num_objects_degraded": 0, 2024-05-08T16:50:32.723 INFO:tasks.workunit.client.0.smithi172.stdout: "num_objects_misplaced": 0, 2024-05-08T16:50:32.723 INFO:tasks.workunit.client.0.smithi172.stdout: "num_objects_unfound": 0, 2024-05-08T16:50:32.723 INFO:tasks.workunit.client.0.smithi172.stdout: "num_objects_dirty": 1, 2024-05-08T16:50:32.723 INFO:tasks.workunit.client.0.smithi172.stdout: "num_whiteouts": 0, 2024-05-08T16:50:32.723 INFO:tasks.workunit.client.0.smithi172.stdout: "num_read": 0, 2024-05-08T16:50:32.723 INFO:tasks.workunit.client.0.smithi172.stdout: "num_read_kb": 0, 2024-05-08T16:50:32.723 INFO:tasks.workunit.client.0.smithi172.stdout: "num_write": 1, 2024-05-08T16:50:32.723 INFO:tasks.workunit.client.0.smithi172.stdout: "num_write_kb": 4, 2024-05-08T16:50:32.723 INFO:tasks.workunit.client.0.smithi172.stdout: "num_scrub_errors": 0, 2024-05-08T16:50:32.723 INFO:tasks.workunit.client.0.smithi172.stdout: "num_shallow_scrub_errors": 0, 2024-05-08T16:50:32.724 INFO:tasks.workunit.client.0.smithi172.stdout: "num_deep_scrub_errors": 0, 2024-05-08T16:50:32.724 INFO:tasks.workunit.client.0.smithi172.stdout: "num_objects_recovered": 0, 2024-05-08T16:50:32.724 INFO:tasks.workunit.client.0.smithi172.stdout: "num_bytes_recovered": 0, 2024-05-08T16:50:32.724 INFO:tasks.workunit.client.0.smithi172.stdout: "num_keys_recovered": 0, 2024-05-08T16:50:32.724 INFO:tasks.workunit.client.0.smithi172.stdout: "num_objects_omap": 0, 2024-05-08T16:50:32.724 INFO:tasks.workunit.client.0.smithi172.stdout: "num_objects_hit_set_archive": 0, 2024-05-08T16:50:32.724 INFO:tasks.workunit.client.0.smithi172.stdout: "num_bytes_hit_set_archive": 0, 2024-05-08T16:50:32.724 INFO:tasks.workunit.client.0.smithi172.stdout: "num_flush": 0, 2024-05-08T16:50:32.724 INFO:tasks.workunit.client.0.smithi172.stdout: "num_flush_kb": 0, 2024-05-08T16:50:32.724 INFO:tasks.workunit.client.0.smithi172.stdout: "num_evict": 0, 2024-05-08T16:50:32.724 INFO:tasks.workunit.client.0.smithi172.stdout: "num_evict_kb": 0, 2024-05-08T16:50:32.724 INFO:tasks.workunit.client.0.smithi172.stdout: "num_promote": 0, 2024-05-08T16:50:32.724 INFO:tasks.workunit.client.0.smithi172.stdout: "num_flush_mode_high": 0, 2024-05-08T16:50:32.724 INFO:tasks.workunit.client.0.smithi172.stdout: "num_flush_mode_low": 0, 2024-05-08T16:50:32.724 INFO:tasks.workunit.client.0.smithi172.stdout: "num_evict_mode_some": 0, 2024-05-08T16:50:32.725 INFO:tasks.workunit.client.0.smithi172.stdout: "num_evict_mode_full": 0, 2024-05-08T16:50:32.725 INFO:tasks.workunit.client.0.smithi172.stdout: "num_objects_pinned": 0, 2024-05-08T16:50:32.725 INFO:tasks.workunit.client.0.smithi172.stdout: "num_legacy_snapsets": 0, 2024-05-08T16:50:32.725 INFO:tasks.workunit.client.0.smithi172.stdout: "num_large_omap_objects": 0, 2024-05-08T16:50:32.725 INFO:tasks.workunit.client.0.smithi172.stdout: "num_objects_manifest": 0, 2024-05-08T16:50:32.725 INFO:tasks.workunit.client.0.smithi172.stdout: "num_omap_bytes": 0, 2024-05-08T16:50:32.725 INFO:tasks.workunit.client.0.smithi172.stdout: "num_omap_keys": 0, 2024-05-08T16:50:32.725 INFO:tasks.workunit.client.0.smithi172.stdout: "num_objects_repaired": 0 2024-05-08T16:50:32.725 INFO:tasks.workunit.client.0.smithi172.stdout: }, 2024-05-08T16:50:32.725 INFO:tasks.workunit.client.0.smithi172.stdout: "up": [ 2024-05-08T16:50:32.725 INFO:tasks.workunit.client.0.smithi172.stdout: 3, 2024-05-08T16:50:32.725 INFO:tasks.workunit.client.0.smithi172.stdout: 1, 2024-05-08T16:50:32.725 INFO:tasks.workunit.client.0.smithi172.stdout: 4, 2024-05-08T16:50:32.725 INFO:tasks.workunit.client.0.smithi172.stdout: 0, 2024-05-08T16:50:32.725 INFO:tasks.workunit.client.0.smithi172.stdout: 2 2024-05-08T16:50:32.725 INFO:tasks.workunit.client.0.smithi172.stdout: ], 2024-05-08T16:50:32.726 INFO:tasks.workunit.client.0.smithi172.stdout: "acting": [ 2024-05-08T16:50:32.726 INFO:tasks.workunit.client.0.smithi172.stdout: 3, 2024-05-08T16:50:32.726 INFO:tasks.workunit.client.0.smithi172.stdout: 1, 2024-05-08T16:50:32.726 INFO:tasks.workunit.client.0.smithi172.stdout: 4, 2024-05-08T16:50:32.726 INFO:tasks.workunit.client.0.smithi172.stdout: 0, 2024-05-08T16:50:32.726 INFO:tasks.workunit.client.0.smithi172.stdout: 2 2024-05-08T16:50:32.726 INFO:tasks.workunit.client.0.smithi172.stdout: ], 2024-05-08T16:50:32.726 INFO:tasks.workunit.client.0.smithi172.stdout: "avail_no_missing": [], 2024-05-08T16:50:32.726 INFO:tasks.workunit.client.0.smithi172.stdout: "object_location_counts": [], 2024-05-08T16:50:32.726 INFO:tasks.workunit.client.0.smithi172.stdout: "blocked_by": [], 2024-05-08T16:50:32.726 INFO:tasks.workunit.client.0.smithi172.stdout: "up_primary": 3, 2024-05-08T16:50:32.726 INFO:tasks.workunit.client.0.smithi172.stdout: "acting_primary": 3, 2024-05-08T16:50:32.726 INFO:tasks.workunit.client.0.smithi172.stdout: "purged_snaps": [] 2024-05-08T16:50:32.726 INFO:tasks.workunit.client.0.smithi172.stdout: }, 2024-05-08T16:50:32.727 INFO:tasks.workunit.client.0.smithi172.stdout: "empty": 0, 2024-05-08T16:50:32.727 INFO:tasks.workunit.client.0.smithi172.stdout: "dne": 0, 2024-05-08T16:50:32.727 INFO:tasks.workunit.client.0.smithi172.stdout: "incomplete": 0, 2024-05-08T16:50:32.727 INFO:tasks.workunit.client.0.smithi172.stdout: "last_epoch_started": 58, 2024-05-08T16:50:32.727 INFO:tasks.workunit.client.0.smithi172.stdout: "hit_set_history": { 2024-05-08T16:50:32.727 INFO:tasks.workunit.client.0.smithi172.stdout: "current_last_update": "0'0", 2024-05-08T16:50:32.727 INFO:tasks.workunit.client.0.smithi172.stdout: "history": [] 2024-05-08T16:50:32.727 INFO:tasks.workunit.client.0.smithi172.stdout: } 2024-05-08T16:50:32.727 INFO:tasks.workunit.client.0.smithi172.stdout: }, 2024-05-08T16:50:32.727 INFO:tasks.workunit.client.0.smithi172.stdout: { 2024-05-08T16:50:32.727 INFO:tasks.workunit.client.0.smithi172.stdout: "peer": "4(2)", 2024-05-08T16:50:32.727 INFO:tasks.workunit.client.0.smithi172.stdout: "pgid": "2.0s2", 2024-05-08T16:50:32.727 INFO:tasks.workunit.client.0.smithi172.stdout: "shared": "2", 2024-05-08T16:50:32.727 INFO:tasks.workunit.client.0.smithi172.stdout: "last_update": "51'101", 2024-05-08T16:50:32.728 INFO:tasks.workunit.client.0.smithi172.stdout: "last_complete": "51'101", 2024-05-08T16:50:32.728 INFO:tasks.workunit.client.0.smithi172.stdout: "log_tail": "0'0", 2024-05-08T16:50:32.728 INFO:tasks.workunit.client.0.smithi172.stdout: "last_user_version": 101, 2024-05-08T16:50:32.728 INFO:tasks.workunit.client.0.smithi172.stdout: "last_backfill": "MAX", 2024-05-08T16:50:32.728 INFO:tasks.workunit.client.0.smithi172.stdout: "purged_snaps": [], 2024-05-08T16:50:32.728 INFO:tasks.workunit.client.0.smithi172.stdout: "history": { 2024-05-08T16:50:32.728 INFO:tasks.workunit.client.0.smithi172.stdout: "epoch_created": 43, 2024-05-08T16:50:32.728 INFO:tasks.workunit.client.0.smithi172.stdout: "epoch_pool_created": 43, 2024-05-08T16:50:32.728 INFO:tasks.workunit.client.0.smithi172.stdout: "last_epoch_started": 58, 2024-05-08T16:50:32.728 INFO:tasks.workunit.client.0.smithi172.stdout: "last_interval_started": 55, 2024-05-08T16:50:32.728 INFO:tasks.workunit.client.0.smithi172.stdout: "last_epoch_clean": 44, 2024-05-08T16:50:32.728 INFO:tasks.workunit.client.0.smithi172.stdout: "last_interval_clean": 43, 2024-05-08T16:50:32.728 INFO:tasks.workunit.client.0.smithi172.stdout: "last_epoch_split": 0, 2024-05-08T16:50:32.728 INFO:tasks.workunit.client.0.smithi172.stdout: "last_epoch_marked_full": 0, 2024-05-08T16:50:32.728 INFO:tasks.workunit.client.0.smithi172.stdout: "same_up_since": 55, 2024-05-08T16:50:32.728 INFO:tasks.workunit.client.0.smithi172.stdout: "same_interval_since": 55, 2024-05-08T16:50:32.728 INFO:tasks.workunit.client.0.smithi172.stdout: "same_primary_since": 43, 2024-05-08T16:50:32.729 INFO:tasks.workunit.client.0.smithi172.stdout: "last_scrub": "0'0", 2024-05-08T16:50:32.729 INFO:tasks.workunit.client.0.smithi172.stdout: "last_scrub_stamp": "2024-05-08T16:49:25.965607+0000", 2024-05-08T16:50:32.729 INFO:tasks.workunit.client.0.smithi172.stdout: "last_deep_scrub": "0'0", 2024-05-08T16:50:32.729 INFO:tasks.workunit.client.0.smithi172.stdout: "last_deep_scrub_stamp": "2024-05-08T16:49:25.965607+0000", 2024-05-08T16:50:32.729 INFO:tasks.workunit.client.0.smithi172.stdout: "last_clean_scrub_stamp": "2024-05-08T16:49:25.965607+0000", 2024-05-08T16:50:32.729 INFO:tasks.workunit.client.0.smithi172.stdout: "prior_readable_until_ub": 0 2024-05-08T16:50:32.729 INFO:tasks.workunit.client.0.smithi172.stdout: }, 2024-05-08T16:50:32.729 INFO:tasks.workunit.client.0.smithi172.stdout: "stats": { 2024-05-08T16:50:32.729 INFO:tasks.workunit.client.0.smithi172.stdout: "version": "51'101", 2024-05-08T16:50:32.729 INFO:tasks.workunit.client.0.smithi172.stdout: "reported_seq": 133, 2024-05-08T16:50:32.729 INFO:tasks.workunit.client.0.smithi172.stdout: "reported_epoch": 51, 2024-05-08T16:50:32.729 INFO:tasks.workunit.client.0.smithi172.stdout: "state": "active+undersized+degraded", 2024-05-08T16:50:32.729 INFO:tasks.workunit.client.0.smithi172.stdout: "last_fresh": "2024-05-08T16:50:08.881951+0000", 2024-05-08T16:50:32.729 INFO:tasks.workunit.client.0.smithi172.stdout: "last_change": "2024-05-08T16:49:35.993333+0000", 2024-05-08T16:50:32.730 INFO:tasks.workunit.client.0.smithi172.stdout: "last_active": "2024-05-08T16:50:08.881951+0000", 2024-05-08T16:50:32.730 INFO:tasks.workunit.client.0.smithi172.stdout: "last_peered": "2024-05-08T16:50:08.881951+0000", 2024-05-08T16:50:32.730 INFO:tasks.workunit.client.0.smithi172.stdout: "last_clean": "2024-05-08T16:49:33.679662+0000", 2024-05-08T16:50:32.730 INFO:tasks.workunit.client.0.smithi172.stdout: "last_became_active": "2024-05-08T16:49:35.993333+0000", 2024-05-08T16:50:32.730 INFO:tasks.workunit.client.0.smithi172.stdout: "last_became_peered": "2024-05-08T16:49:35.993333+0000", 2024-05-08T16:50:32.730 INFO:tasks.workunit.client.0.smithi172.stdout: "last_unstale": "2024-05-08T16:50:08.881951+0000", 2024-05-08T16:50:32.730 INFO:tasks.workunit.client.0.smithi172.stdout: "last_undegraded": "2024-05-08T16:49:35.587590+0000", 2024-05-08T16:50:32.730 INFO:tasks.workunit.client.0.smithi172.stdout: "last_fullsized": "2024-05-08T16:49:35.587392+0000", 2024-05-08T16:50:32.730 INFO:tasks.workunit.client.0.smithi172.stdout: "mapping_epoch": 55, 2024-05-08T16:50:32.730 INFO:tasks.workunit.client.0.smithi172.stdout: "log_start": "0'0", 2024-05-08T16:50:32.730 INFO:tasks.workunit.client.0.smithi172.stdout: "ondisk_log_start": "0'0", 2024-05-08T16:50:32.730 INFO:tasks.workunit.client.0.smithi172.stdout: "created": 43, 2024-05-08T16:50:32.730 INFO:tasks.workunit.client.0.smithi172.stdout: "last_epoch_clean": 44, 2024-05-08T16:50:32.730 INFO:tasks.workunit.client.0.smithi172.stdout: "parent": "0.0", 2024-05-08T16:50:32.731 INFO:tasks.workunit.client.0.smithi172.stdout: "parent_split_bits": 0, 2024-05-08T16:50:32.731 INFO:tasks.workunit.client.0.smithi172.stdout: "last_scrub": "0'0", 2024-05-08T16:50:32.731 INFO:tasks.workunit.client.0.smithi172.stdout: "last_scrub_stamp": "2024-05-08T16:49:25.965607+0000", 2024-05-08T16:50:32.731 INFO:tasks.workunit.client.0.smithi172.stdout: "last_deep_scrub": "0'0", 2024-05-08T16:50:32.731 INFO:tasks.workunit.client.0.smithi172.stdout: "last_deep_scrub_stamp": "2024-05-08T16:49:25.965607+0000", 2024-05-08T16:50:32.731 INFO:tasks.workunit.client.0.smithi172.stdout: "last_clean_scrub_stamp": "2024-05-08T16:49:25.965607+0000", 2024-05-08T16:50:32.731 INFO:tasks.workunit.client.0.smithi172.stdout: "objects_scrubbed": 0, 2024-05-08T16:50:32.731 INFO:tasks.workunit.client.0.smithi172.stdout: "log_size": 101, 2024-05-08T16:50:32.731 INFO:tasks.workunit.client.0.smithi172.stdout: "log_dups_size": 0, 2024-05-08T16:50:32.731 INFO:tasks.workunit.client.0.smithi172.stdout: "ondisk_log_size": 101, 2024-05-08T16:50:32.731 INFO:tasks.workunit.client.0.smithi172.stdout: "stats_invalid": false, 2024-05-08T16:50:32.731 INFO:tasks.workunit.client.0.smithi172.stdout: "dirty_stats_invalid": false, 2024-05-08T16:50:32.731 INFO:tasks.workunit.client.0.smithi172.stdout: "omap_stats_invalid": false, 2024-05-08T16:50:32.731 INFO:tasks.workunit.client.0.smithi172.stdout: "hitset_stats_invalid": false, 2024-05-08T16:50:32.731 INFO:tasks.workunit.client.0.smithi172.stdout: "hitset_bytes_stats_invalid": false, 2024-05-08T16:50:32.732 INFO:tasks.workunit.client.0.smithi172.stdout: "pin_stats_invalid": false, 2024-05-08T16:50:32.732 INFO:tasks.workunit.client.0.smithi172.stdout: "manifest_stats_invalid": false, 2024-05-08T16:50:32.732 INFO:tasks.workunit.client.0.smithi172.stdout: "snaptrimq_len": 0, 2024-05-08T16:50:32.732 INFO:tasks.workunit.client.0.smithi172.stdout: "last_scrub_duration": 0, 2024-05-08T16:50:32.732 INFO:tasks.workunit.client.0.smithi172.stdout: "scrub_schedule": "periodic scrub scheduled @ 2024-05-10T04:08:32.277486+0000", 2024-05-08T16:50:32.732 INFO:tasks.workunit.client.0.smithi172.stdout: "scrub_duration": 0, 2024-05-08T16:50:32.732 INFO:tasks.workunit.client.0.smithi172.stdout: "objects_trimmed": 0, 2024-05-08T16:50:32.732 INFO:tasks.workunit.client.0.smithi172.stdout: "snaptrim_duration": 0, 2024-05-08T16:50:32.732 INFO:tasks.workunit.client.0.smithi172.stdout: "stat_sum": { 2024-05-08T16:50:32.732 INFO:tasks.workunit.client.0.smithi172.stdout: "num_bytes": 413696, 2024-05-08T16:50:32.732 INFO:tasks.workunit.client.0.smithi172.stdout: "num_objects": 101, 2024-05-08T16:50:32.732 INFO:tasks.workunit.client.0.smithi172.stdout: "num_object_clones": 0, 2024-05-08T16:50:32.732 INFO:tasks.workunit.client.0.smithi172.stdout: "num_object_copies": 505, 2024-05-08T16:50:32.732 INFO:tasks.workunit.client.0.smithi172.stdout: "num_objects_missing_on_primary": 0, 2024-05-08T16:50:32.732 INFO:tasks.workunit.client.0.smithi172.stdout: "num_objects_missing": 0, 2024-05-08T16:50:32.733 INFO:tasks.workunit.client.0.smithi172.stdout: "num_objects_degraded": 101, 2024-05-08T16:50:32.733 INFO:tasks.workunit.client.0.smithi172.stdout: "num_objects_misplaced": 0, 2024-05-08T16:50:32.733 INFO:tasks.workunit.client.0.smithi172.stdout: "num_objects_unfound": 0, 2024-05-08T16:50:32.733 INFO:tasks.workunit.client.0.smithi172.stdout: "num_objects_dirty": 101, 2024-05-08T16:50:32.733 INFO:tasks.workunit.client.0.smithi172.stdout: "num_whiteouts": 0, 2024-05-08T16:50:32.733 INFO:tasks.workunit.client.0.smithi172.stdout: "num_read": 0, 2024-05-08T16:50:32.733 INFO:tasks.workunit.client.0.smithi172.stdout: "num_read_kb": 0, 2024-05-08T16:50:32.733 INFO:tasks.workunit.client.0.smithi172.stdout: "num_write": 101, 2024-05-08T16:50:32.733 INFO:tasks.workunit.client.0.smithi172.stdout: "num_write_kb": 404, 2024-05-08T16:50:32.733 INFO:tasks.workunit.client.0.smithi172.stdout: "num_scrub_errors": 0, 2024-05-08T16:50:32.733 INFO:tasks.workunit.client.0.smithi172.stdout: "num_shallow_scrub_errors": 0, 2024-05-08T16:50:32.733 INFO:tasks.workunit.client.0.smithi172.stdout: "num_deep_scrub_errors": 0, 2024-05-08T16:50:32.733 INFO:tasks.workunit.client.0.smithi172.stdout: "num_objects_recovered": 0, 2024-05-08T16:50:32.733 INFO:tasks.workunit.client.0.smithi172.stdout: "num_bytes_recovered": 0, 2024-05-08T16:50:32.733 INFO:tasks.workunit.client.0.smithi172.stdout: "num_keys_recovered": 0, 2024-05-08T16:50:32.733 INFO:tasks.workunit.client.0.smithi172.stdout: "num_objects_omap": 0, 2024-05-08T16:50:32.733 INFO:tasks.workunit.client.0.smithi172.stdout: "num_objects_hit_set_archive": 0, 2024-05-08T16:50:32.734 INFO:tasks.workunit.client.0.smithi172.stdout: "num_bytes_hit_set_archive": 0, 2024-05-08T16:50:32.734 INFO:tasks.workunit.client.0.smithi172.stdout: "num_flush": 0, 2024-05-08T16:50:32.734 INFO:tasks.workunit.client.0.smithi172.stdout: "num_flush_kb": 0, 2024-05-08T16:50:32.734 INFO:tasks.workunit.client.0.smithi172.stdout: "num_evict": 0, 2024-05-08T16:50:32.734 INFO:tasks.workunit.client.0.smithi172.stdout: "num_evict_kb": 0, 2024-05-08T16:50:32.734 INFO:tasks.workunit.client.0.smithi172.stdout: "num_promote": 0, 2024-05-08T16:50:32.734 INFO:tasks.workunit.client.0.smithi172.stdout: "num_flush_mode_high": 0, 2024-05-08T16:50:32.734 INFO:tasks.workunit.client.0.smithi172.stdout: "num_flush_mode_low": 0, 2024-05-08T16:50:32.734 INFO:tasks.workunit.client.0.smithi172.stdout: "num_evict_mode_some": 0, 2024-05-08T16:50:32.734 INFO:tasks.workunit.client.0.smithi172.stdout: "num_evict_mode_full": 0, 2024-05-08T16:50:32.734 INFO:tasks.workunit.client.0.smithi172.stdout: "num_objects_pinned": 0, 2024-05-08T16:50:32.734 INFO:tasks.workunit.client.0.smithi172.stdout: "num_legacy_snapsets": 0, 2024-05-08T16:50:32.734 INFO:tasks.workunit.client.0.smithi172.stdout: "num_large_omap_objects": 0, 2024-05-08T16:50:32.734 INFO:tasks.workunit.client.0.smithi172.stdout: "num_objects_manifest": 0, 2024-05-08T16:50:32.735 INFO:tasks.workunit.client.0.smithi172.stdout: "num_omap_bytes": 0, 2024-05-08T16:50:32.735 INFO:tasks.workunit.client.0.smithi172.stdout: "num_omap_keys": 0, 2024-05-08T16:50:32.735 INFO:tasks.workunit.client.0.smithi172.stdout: "num_objects_repaired": 0 2024-05-08T16:50:32.735 INFO:tasks.workunit.client.0.smithi172.stdout: }, 2024-05-08T16:50:32.735 INFO:tasks.workunit.client.0.smithi172.stdout: "up": [ 2024-05-08T16:50:32.735 INFO:tasks.workunit.client.0.smithi172.stdout: 3, 2024-05-08T16:50:32.735 INFO:tasks.workunit.client.0.smithi172.stdout: 1, 2024-05-08T16:50:32.735 INFO:tasks.workunit.client.0.smithi172.stdout: 4, 2024-05-08T16:50:32.735 INFO:tasks.workunit.client.0.smithi172.stdout: 0, 2024-05-08T16:50:32.735 INFO:tasks.workunit.client.0.smithi172.stdout: 2 2024-05-08T16:50:32.735 INFO:tasks.workunit.client.0.smithi172.stdout: ], 2024-05-08T16:50:32.735 INFO:tasks.workunit.client.0.smithi172.stdout: "acting": [ 2024-05-08T16:50:32.735 INFO:tasks.workunit.client.0.smithi172.stdout: 3, 2024-05-08T16:50:32.735 INFO:tasks.workunit.client.0.smithi172.stdout: 1, 2024-05-08T16:50:32.735 INFO:tasks.workunit.client.0.smithi172.stdout: 4, 2024-05-08T16:50:32.735 INFO:tasks.workunit.client.0.smithi172.stdout: 0, 2024-05-08T16:50:32.736 INFO:tasks.workunit.client.0.smithi172.stdout: 2 2024-05-08T16:50:32.736 INFO:tasks.workunit.client.0.smithi172.stdout: ], 2024-05-08T16:50:32.736 INFO:tasks.workunit.client.0.smithi172.stdout: "avail_no_missing": [ 2024-05-08T16:50:32.736 INFO:tasks.workunit.client.0.smithi172.stdout: "3(0)", 2024-05-08T16:50:32.736 INFO:tasks.workunit.client.0.smithi172.stdout: "0(3)", 2024-05-08T16:50:32.736 INFO:tasks.workunit.client.0.smithi172.stdout: "1(1)", 2024-05-08T16:50:32.736 INFO:tasks.workunit.client.0.smithi172.stdout: "4(2)" 2024-05-08T16:50:32.736 INFO:tasks.workunit.client.0.smithi172.stdout: ], 2024-05-08T16:50:32.736 INFO:tasks.workunit.client.0.smithi172.stdout: "object_location_counts": [ 2024-05-08T16:50:32.736 INFO:tasks.workunit.client.0.smithi172.stdout: { 2024-05-08T16:50:32.736 INFO:tasks.workunit.client.0.smithi172.stdout: "shards": "0(3),1(1),3(0),4(2)", 2024-05-08T16:50:32.736 INFO:tasks.workunit.client.0.smithi172.stdout: "objects": 101 2024-05-08T16:50:32.736 INFO:tasks.workunit.client.0.smithi172.stdout: } 2024-05-08T16:50:32.736 INFO:tasks.workunit.client.0.smithi172.stdout: ], 2024-05-08T16:50:32.737 INFO:tasks.workunit.client.0.smithi172.stdout: "blocked_by": [], 2024-05-08T16:50:32.737 INFO:tasks.workunit.client.0.smithi172.stdout: "up_primary": 3, 2024-05-08T16:50:32.737 INFO:tasks.workunit.client.0.smithi172.stdout: "acting_primary": 3, 2024-05-08T16:50:32.737 INFO:tasks.workunit.client.0.smithi172.stdout: "purged_snaps": [] 2024-05-08T16:50:32.737 INFO:tasks.workunit.client.0.smithi172.stdout: }, 2024-05-08T16:50:32.737 INFO:tasks.workunit.client.0.smithi172.stdout: "empty": 0, 2024-05-08T16:50:32.737 INFO:tasks.workunit.client.0.smithi172.stdout: "dne": 0, 2024-05-08T16:50:32.737 INFO:tasks.workunit.client.0.smithi172.stdout: "incomplete": 0, 2024-05-08T16:50:32.737 INFO:tasks.workunit.client.0.smithi172.stdout: "last_epoch_started": 58, 2024-05-08T16:50:32.737 INFO:tasks.workunit.client.0.smithi172.stdout: "hit_set_history": { 2024-05-08T16:50:32.737 INFO:tasks.workunit.client.0.smithi172.stdout: "current_last_update": "0'0", 2024-05-08T16:50:32.737 INFO:tasks.workunit.client.0.smithi172.stdout: "history": [] 2024-05-08T16:50:32.737 INFO:tasks.workunit.client.0.smithi172.stdout: } 2024-05-08T16:50:32.737 INFO:tasks.workunit.client.0.smithi172.stdout: } 2024-05-08T16:50:32.738 INFO:tasks.workunit.client.0.smithi172.stdout: ], 2024-05-08T16:50:32.738 INFO:tasks.workunit.client.0.smithi172.stdout: "recovery_state": [ 2024-05-08T16:50:32.738 INFO:tasks.workunit.client.0.smithi172.stdout: { 2024-05-08T16:50:32.738 INFO:tasks.workunit.client.0.smithi172.stdout: "name": "Started/Primary/Active", 2024-05-08T16:50:32.738 INFO:tasks.workunit.client.0.smithi172.stdout: "enter_time": "2024-05-08T16:50:17.986985+0000", 2024-05-08T16:50:32.738 INFO:tasks.workunit.client.0.smithi172.stdout: "might_have_unfound": [ 2024-05-08T16:50:32.738 INFO:tasks.workunit.client.0.smithi172.stdout: { 2024-05-08T16:50:32.738 INFO:tasks.workunit.client.0.smithi172.stdout: "osd": "0(3)", 2024-05-08T16:50:32.738 INFO:tasks.workunit.client.0.smithi172.stdout: "status": "already probed" 2024-05-08T16:50:32.738 INFO:tasks.workunit.client.0.smithi172.stdout: }, 2024-05-08T16:50:32.738 INFO:tasks.workunit.client.0.smithi172.stdout: { 2024-05-08T16:50:32.738 INFO:tasks.workunit.client.0.smithi172.stdout: "osd": "1(1)", 2024-05-08T16:50:32.739 INFO:tasks.workunit.client.0.smithi172.stdout: "status": "already probed" 2024-05-08T16:50:32.739 INFO:tasks.workunit.client.0.smithi172.stdout: }, 2024-05-08T16:50:32.739 INFO:tasks.workunit.client.0.smithi172.stdout: { 2024-05-08T16:50:32.739 INFO:tasks.workunit.client.0.smithi172.stdout: "osd": "2(4)", 2024-05-08T16:50:32.739 INFO:tasks.workunit.client.0.smithi172.stdout: "status": "already probed" 2024-05-08T16:50:32.739 INFO:tasks.workunit.client.0.smithi172.stdout: }, 2024-05-08T16:50:32.739 INFO:tasks.workunit.client.0.smithi172.stdout: { 2024-05-08T16:50:32.739 INFO:tasks.workunit.client.0.smithi172.stdout: "osd": "4(2)", 2024-05-08T16:50:32.739 INFO:tasks.workunit.client.0.smithi172.stdout: "status": "already probed" 2024-05-08T16:50:32.739 INFO:tasks.workunit.client.0.smithi172.stdout: } 2024-05-08T16:50:32.739 INFO:tasks.workunit.client.0.smithi172.stdout: ], 2024-05-08T16:50:32.739 INFO:tasks.workunit.client.0.smithi172.stdout: "recovery_progress": { 2024-05-08T16:50:32.739 INFO:tasks.workunit.client.0.smithi172.stdout: "backfill_targets": [], 2024-05-08T16:50:32.739 INFO:tasks.workunit.client.0.smithi172.stdout: "waiting_on_backfill": [], 2024-05-08T16:50:32.739 INFO:tasks.workunit.client.0.smithi172.stdout: "last_backfill_started": "MIN", 2024-05-08T16:50:32.740 INFO:tasks.workunit.client.0.smithi172.stdout: "backfill_info": { 2024-05-08T16:50:32.740 INFO:tasks.workunit.client.0.smithi172.stdout: "begin": "MIN", 2024-05-08T16:50:32.740 INFO:tasks.workunit.client.0.smithi172.stdout: "end": "MIN", 2024-05-08T16:50:32.740 INFO:tasks.workunit.client.0.smithi172.stdout: "objects": [] 2024-05-08T16:50:32.740 INFO:tasks.workunit.client.0.smithi172.stdout: }, 2024-05-08T16:50:32.740 INFO:tasks.workunit.client.0.smithi172.stdout: "peer_backfill_info": [], 2024-05-08T16:50:32.740 INFO:tasks.workunit.client.0.smithi172.stdout: "backfills_in_flight": [], 2024-05-08T16:50:32.740 INFO:tasks.workunit.client.0.smithi172.stdout: "recovering": [], 2024-05-08T16:50:32.740 INFO:tasks.workunit.client.0.smithi172.stdout: "pg_backend": { 2024-05-08T16:50:32.740 INFO:tasks.workunit.client.0.smithi172.stdout: "recovery_ops": [], 2024-05-08T16:50:32.740 INFO:tasks.workunit.client.0.smithi172.stdout: "read_ops": [] 2024-05-08T16:50:32.740 INFO:tasks.workunit.client.0.smithi172.stdout: } 2024-05-08T16:50:32.740 INFO:tasks.workunit.client.0.smithi172.stdout: } 2024-05-08T16:50:32.740 INFO:tasks.workunit.client.0.smithi172.stdout: }, 2024-05-08T16:50:32.740 INFO:tasks.workunit.client.0.smithi172.stdout: { 2024-05-08T16:50:32.741 INFO:tasks.workunit.client.0.smithi172.stdout: "name": "Started", 2024-05-08T16:50:32.741 INFO:tasks.workunit.client.0.smithi172.stdout: "enter_time": "2024-05-08T16:50:16.805201+0000" 2024-05-08T16:50:32.741 INFO:tasks.workunit.client.0.smithi172.stdout: } 2024-05-08T16:50:32.741 INFO:tasks.workunit.client.0.smithi172.stdout: ], 2024-05-08T16:50:32.741 INFO:tasks.workunit.client.0.smithi172.stdout: "scrubber": { 2024-05-08T16:50:32.741 INFO:tasks.workunit.client.0.smithi172.stdout: "active": false, 2024-05-08T16:50:32.741 INFO:tasks.workunit.client.0.smithi172.stdout: "must_scrub": false, 2024-05-08T16:50:32.741 INFO:tasks.workunit.client.0.smithi172.stdout: "must_deep_scrub": false, 2024-05-08T16:50:32.741 INFO:tasks.workunit.client.0.smithi172.stdout: "must_repair": false, 2024-05-08T16:50:32.741 INFO:tasks.workunit.client.0.smithi172.stdout: "need_auto": false, 2024-05-08T16:50:32.741 INFO:tasks.workunit.client.0.smithi172.stdout: "scrub_reg_stamp": "2024-05-10T04:08:32.277486+0000", 2024-05-08T16:50:32.741 INFO:tasks.workunit.client.0.smithi172.stdout: "schedule": "no scrub is scheduled" 2024-05-08T16:50:32.741 INFO:tasks.workunit.client.0.smithi172.stdout: }, 2024-05-08T16:50:32.741 INFO:tasks.workunit.client.0.smithi172.stdout: "agent_state": {} 2024-05-08T16:50:32.741 INFO:tasks.workunit.client.0.smithi172.stdout:} 2024-05-08T16:50:32.742 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:664: TEST_ec_recovery_unfound: ceph pg 2.0 list_unfound 2024-05-08T16:50:32.742 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:664: TEST_ec_recovery_unfound: grep -q obj75 2024-05-08T16:50:32.839 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:666: TEST_ec_recovery_unfound: ceph pg 2.0 list_unfound 2024-05-08T16:50:32.839 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:666: TEST_ec_recovery_unfound: jq .available_might_have_unfound 2024-05-08T16:50:32.979 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:666: TEST_ec_recovery_unfound: check=true 2024-05-08T16:50:32.979 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:667: TEST_ec_recovery_unfound: test true == true 2024-05-08T16:50:32.980 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:669: TEST_ec_recovery_unfound: ceph pg 2.0 list_unfound 2024-05-08T16:50:32.980 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:669: TEST_ec_recovery_unfound: jq '.might_have_unfound | length' 2024-05-08T16:50:33.121 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:669: TEST_ec_recovery_unfound: check=0 2024-05-08T16:50:33.121 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:670: TEST_ec_recovery_unfound: test 0 == 0 2024-05-08T16:50:33.121 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:673: TEST_ec_recovery_unfound: timeout 5 rados -p pool-jerasure get obj75 td/test-erasure-eio/CHECK 2024-05-08T16:50:38.124 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:674: TEST_ec_recovery_unfound: test 124 = 124 2024-05-08T16:50:38.124 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:676: TEST_ec_recovery_unfound: ceph pg 2.0 mark_unfound_lost delete 2024-05-08T16:50:38.447 INFO:tasks.workunit.client.0.smithi172.stderr:pg has no unfound objects 2024-05-08T16:50:38.472 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:678: TEST_ec_recovery_unfound: wait_for_clean 2024-05-08T16:50:38.472 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-05-08T16:50:38.472 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-05-08T16:50:38.472 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-05-08T16:50:38.473 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-05-08T16:50:38.474 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-05-08T16:50:38.474 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-05-08T16:50:38.474 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-05-08T16:50:38.474 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-05-08T16:50:38.474 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-05-08T16:50:38.600 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-05-08T16:50:38.600 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-05-08T16:50:38.600 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-05-08T16:50:38.600 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-05-08T16:50:38.600 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-05-08T16:50:38.600 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-05-08T16:50:39.042 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-05-08T16:50:39.042 INFO:tasks.workunit.client.0.smithi172.stderr:1 2024-05-08T16:50:39.042 INFO:tasks.workunit.client.0.smithi172.stderr:2 2024-05-08T16:50:39.043 INFO:tasks.workunit.client.0.smithi172.stderr:3 2024-05-08T16:50:39.043 INFO:tasks.workunit.client.0.smithi172.stderr:4' 2024-05-08T16:50:39.043 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-05-08T16:50:39.043 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T16:50:39.043 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-05-08T16:50:39.196 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=25769803834 2024-05-08T16:50:39.197 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 25769803834 2024-05-08T16:50:39.197 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803834' 2024-05-08T16:50:39.197 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T16:50:39.197 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-05-08T16:50:39.347 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=55834574898 2024-05-08T16:50:39.347 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 55834574898 2024-05-08T16:50:39.347 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803834 1-55834574898' 2024-05-08T16:50:39.347 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T16:50:39.347 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-05-08T16:50:39.491 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=227633266695 2024-05-08T16:50:39.491 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 227633266695 2024-05-08T16:50:39.491 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803834 1-55834574898 2-227633266695' 2024-05-08T16:50:39.491 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T16:50:39.491 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-05-08T16:50:39.634 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=115964117025 2024-05-08T16:50:39.634 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 115964117025 2024-05-08T16:50:39.635 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803834 1-55834574898 2-227633266695 3-115964117025' 2024-05-08T16:50:39.635 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T16:50:39.635 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-05-08T16:50:39.783 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=163208757271 2024-05-08T16:50:39.783 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 163208757271 2024-05-08T16:50:39.783 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803834 1-55834574898 2-227633266695 3-115964117025 4-163208757271' 2024-05-08T16:50:39.783 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-08T16:50:39.784 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-25769803834 2024-05-08T16:50:39.784 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-08T16:50:39.784 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-05-08T16:50:39.785 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-25769803834 2024-05-08T16:50:39.785 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-08T16:50:39.786 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=25769803834 2024-05-08T16:50:39.786 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 25769803834' 2024-05-08T16:50:39.787 INFO:tasks.workunit.client.0.smithi172.stdout:waiting osd.0 seq 25769803834 2024-05-08T16:50:39.787 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-05-08T16:50:40.207 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 25769803835 -lt 25769803834 2024-05-08T16:50:40.208 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-08T16:50:40.208 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-55834574898 2024-05-08T16:50:40.208 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-08T16:50:40.209 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-05-08T16:50:40.210 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-55834574898 2024-05-08T16:50:40.210 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-08T16:50:40.212 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=55834574898 2024-05-08T16:50:40.212 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 55834574898' 2024-05-08T16:50:40.212 INFO:tasks.workunit.client.0.smithi172.stdout:waiting osd.1 seq 55834574898 2024-05-08T16:50:40.212 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-05-08T16:50:40.687 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 55834574898 -lt 55834574898 2024-05-08T16:50:40.687 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-08T16:50:40.688 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-227633266695 2024-05-08T16:50:40.688 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-08T16:50:40.690 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-05-08T16:50:40.690 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-227633266695 2024-05-08T16:50:40.691 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-08T16:50:40.692 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=227633266695 2024-05-08T16:50:40.692 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 227633266695' 2024-05-08T16:50:40.692 INFO:tasks.workunit.client.0.smithi172.stdout:waiting osd.2 seq 227633266695 2024-05-08T16:50:40.692 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-05-08T16:50:41.127 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 227633266695 -lt 227633266695 2024-05-08T16:50:41.128 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-08T16:50:41.128 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-115964117025 2024-05-08T16:50:41.128 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-08T16:50:41.129 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-05-08T16:50:41.130 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-115964117025 2024-05-08T16:50:41.130 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-08T16:50:41.131 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=115964117025 2024-05-08T16:50:41.132 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 115964117025' 2024-05-08T16:50:41.132 INFO:tasks.workunit.client.0.smithi172.stdout:waiting osd.3 seq 115964117025 2024-05-08T16:50:41.132 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-05-08T16:50:41.513 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 115964117025 -lt 115964117025 2024-05-08T16:50:41.513 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-08T16:50:41.514 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-163208757271 2024-05-08T16:50:41.514 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-08T16:50:41.516 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-05-08T16:50:41.517 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-163208757271 2024-05-08T16:50:41.517 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-08T16:50:41.518 INFO:tasks.workunit.client.0.smithi172.stdout:waiting osd.4 seq 163208757271 2024-05-08T16:50:41.519 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=163208757271 2024-05-08T16:50:41.519 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 163208757271' 2024-05-08T16:50:41.519 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-05-08T16:50:41.939 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 163208757271 -lt 163208757271 2024-05-08T16:50:41.940 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-05-08T16:50:41.940 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-05-08T16:50:41.941 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-05-08T16:50:42.489 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-05-08T16:50:42.489 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-05-08T16:50:42.489 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-05-08T16:50:42.490 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-05-08T16:50:42.490 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-05-08T16:50:42.490 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-05-08T16:50:42.490 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-05-08T16:50:42.490 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-05-08T16:50:42.873 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-05-08T16:50:42.874 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-05-08T16:50:42.874 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-05-08T16:50:42.874 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-05-08T16:50:43.386 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-05-08T16:50:43.386 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-05-08T16:50:43.387 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-05-08T16:50:43.387 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: seq 1 100 2024-05-08T16:50:43.389 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:50:43.389 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj1 = obj75 ']' 2024-05-08T16:50:43.389 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj1 td/test-erasure-eio/CHECK 2024-05-08T16:50:43.428 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:50:43.429 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:50:43.429 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj2 = obj75 ']' 2024-05-08T16:50:43.429 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj2 td/test-erasure-eio/CHECK 2024-05-08T16:50:43.503 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:50:43.504 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:50:43.505 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj3 = obj75 ']' 2024-05-08T16:50:43.505 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj3 td/test-erasure-eio/CHECK 2024-05-08T16:50:43.536 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:50:43.537 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:50:43.537 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj4 = obj75 ']' 2024-05-08T16:50:43.537 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj4 td/test-erasure-eio/CHECK 2024-05-08T16:50:43.565 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:50:43.566 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:50:43.566 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj5 = obj75 ']' 2024-05-08T16:50:43.566 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj5 td/test-erasure-eio/CHECK 2024-05-08T16:50:43.604 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:50:43.605 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:50:43.605 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj6 = obj75 ']' 2024-05-08T16:50:43.605 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj6 td/test-erasure-eio/CHECK 2024-05-08T16:50:43.633 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:50:43.634 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:50:43.634 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj7 = obj75 ']' 2024-05-08T16:50:43.634 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj7 td/test-erasure-eio/CHECK 2024-05-08T16:50:43.662 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:50:43.663 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:50:43.663 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj8 = obj75 ']' 2024-05-08T16:50:43.663 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj8 td/test-erasure-eio/CHECK 2024-05-08T16:50:43.693 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:50:43.694 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:50:43.694 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj9 = obj75 ']' 2024-05-08T16:50:43.694 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj9 td/test-erasure-eio/CHECK 2024-05-08T16:50:43.722 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:50:43.723 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:50:43.723 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj10 = obj75 ']' 2024-05-08T16:50:43.723 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj10 td/test-erasure-eio/CHECK 2024-05-08T16:50:43.751 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:50:43.752 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:50:43.752 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj11 = obj75 ']' 2024-05-08T16:50:43.752 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj11 td/test-erasure-eio/CHECK 2024-05-08T16:50:43.780 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:50:43.781 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:50:43.781 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj12 = obj75 ']' 2024-05-08T16:50:43.781 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj12 td/test-erasure-eio/CHECK 2024-05-08T16:50:43.862 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:50:43.863 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:50:43.863 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj13 = obj75 ']' 2024-05-08T16:50:43.863 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj13 td/test-erasure-eio/CHECK 2024-05-08T16:50:43.892 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:50:43.893 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:50:43.893 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj14 = obj75 ']' 2024-05-08T16:50:43.893 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj14 td/test-erasure-eio/CHECK 2024-05-08T16:50:43.922 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:50:43.923 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:50:43.923 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj15 = obj75 ']' 2024-05-08T16:50:43.923 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj15 td/test-erasure-eio/CHECK 2024-05-08T16:50:43.951 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:50:43.952 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:50:43.952 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj16 = obj75 ']' 2024-05-08T16:50:43.952 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj16 td/test-erasure-eio/CHECK 2024-05-08T16:50:43.980 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:50:43.981 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:50:43.981 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj17 = obj75 ']' 2024-05-08T16:50:43.981 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj17 td/test-erasure-eio/CHECK 2024-05-08T16:50:44.010 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:50:44.010 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:50:44.011 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj18 = obj75 ']' 2024-05-08T16:50:44.011 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj18 td/test-erasure-eio/CHECK 2024-05-08T16:50:44.040 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:50:44.041 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:50:44.041 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj19 = obj75 ']' 2024-05-08T16:50:44.041 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj19 td/test-erasure-eio/CHECK 2024-05-08T16:50:44.070 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:50:44.071 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:50:44.071 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj20 = obj75 ']' 2024-05-08T16:50:44.071 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj20 td/test-erasure-eio/CHECK 2024-05-08T16:50:44.099 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:50:44.100 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:50:44.100 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj21 = obj75 ']' 2024-05-08T16:50:44.100 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj21 td/test-erasure-eio/CHECK 2024-05-08T16:50:44.128 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:50:44.129 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:50:44.129 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj22 = obj75 ']' 2024-05-08T16:50:44.129 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj22 td/test-erasure-eio/CHECK 2024-05-08T16:50:44.157 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:50:44.158 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:50:44.158 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj23 = obj75 ']' 2024-05-08T16:50:44.158 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj23 td/test-erasure-eio/CHECK 2024-05-08T16:50:44.186 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:50:44.187 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:50:44.187 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj24 = obj75 ']' 2024-05-08T16:50:44.187 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj24 td/test-erasure-eio/CHECK 2024-05-08T16:50:44.216 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:50:44.217 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:50:44.217 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj25 = obj75 ']' 2024-05-08T16:50:44.217 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj25 td/test-erasure-eio/CHECK 2024-05-08T16:50:44.271 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:50:44.272 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:50:44.272 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj26 = obj75 ']' 2024-05-08T16:50:44.272 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj26 td/test-erasure-eio/CHECK 2024-05-08T16:50:44.300 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:50:44.301 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:50:44.301 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj27 = obj75 ']' 2024-05-08T16:50:44.301 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj27 td/test-erasure-eio/CHECK 2024-05-08T16:50:44.381 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:50:44.382 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:50:44.382 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj28 = obj75 ']' 2024-05-08T16:50:44.382 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj28 td/test-erasure-eio/CHECK 2024-05-08T16:50:44.411 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:50:44.412 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:50:44.412 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj29 = obj75 ']' 2024-05-08T16:50:44.412 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj29 td/test-erasure-eio/CHECK 2024-05-08T16:50:44.440 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:50:44.441 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:50:44.441 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj30 = obj75 ']' 2024-05-08T16:50:44.441 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj30 td/test-erasure-eio/CHECK 2024-05-08T16:50:44.469 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:50:44.470 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:50:44.470 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj31 = obj75 ']' 2024-05-08T16:50:44.471 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj31 td/test-erasure-eio/CHECK 2024-05-08T16:50:44.498 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:50:44.499 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:50:44.499 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj32 = obj75 ']' 2024-05-08T16:50:44.500 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj32 td/test-erasure-eio/CHECK 2024-05-08T16:50:44.528 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:50:44.529 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:50:44.529 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj33 = obj75 ']' 2024-05-08T16:50:44.529 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj33 td/test-erasure-eio/CHECK 2024-05-08T16:50:44.557 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:50:44.558 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:50:44.558 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj34 = obj75 ']' 2024-05-08T16:50:44.559 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj34 td/test-erasure-eio/CHECK 2024-05-08T16:50:44.587 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:50:44.587 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:50:44.587 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj35 = obj75 ']' 2024-05-08T16:50:44.587 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj35 td/test-erasure-eio/CHECK 2024-05-08T16:50:44.616 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:50:44.616 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:50:44.617 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj36 = obj75 ']' 2024-05-08T16:50:44.617 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj36 td/test-erasure-eio/CHECK 2024-05-08T16:50:44.645 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:50:44.645 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:50:44.646 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj37 = obj75 ']' 2024-05-08T16:50:44.646 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj37 td/test-erasure-eio/CHECK 2024-05-08T16:50:44.674 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:50:44.675 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:50:44.675 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj38 = obj75 ']' 2024-05-08T16:50:44.675 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj38 td/test-erasure-eio/CHECK 2024-05-08T16:50:44.703 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:50:44.704 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:50:44.704 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj39 = obj75 ']' 2024-05-08T16:50:44.704 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj39 td/test-erasure-eio/CHECK 2024-05-08T16:50:44.732 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:50:44.733 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:50:44.733 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj40 = obj75 ']' 2024-05-08T16:50:44.733 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj40 td/test-erasure-eio/CHECK 2024-05-08T16:50:44.761 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:50:44.762 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:50:44.762 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj41 = obj75 ']' 2024-05-08T16:50:44.762 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj41 td/test-erasure-eio/CHECK 2024-05-08T16:50:44.790 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:50:44.791 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:50:44.791 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj42 = obj75 ']' 2024-05-08T16:50:44.791 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj42 td/test-erasure-eio/CHECK 2024-05-08T16:50:44.819 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:50:44.821 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:50:44.821 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj43 = obj75 ']' 2024-05-08T16:50:44.821 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj43 td/test-erasure-eio/CHECK 2024-05-08T16:50:44.848 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:50:44.849 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:50:44.849 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj44 = obj75 ']' 2024-05-08T16:50:44.849 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj44 td/test-erasure-eio/CHECK 2024-05-08T16:50:44.878 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:50:44.879 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:50:44.879 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj45 = obj75 ']' 2024-05-08T16:50:44.879 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj45 td/test-erasure-eio/CHECK 2024-05-08T16:50:44.907 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:50:44.908 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:50:44.908 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj46 = obj75 ']' 2024-05-08T16:50:44.908 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj46 td/test-erasure-eio/CHECK 2024-05-08T16:50:44.937 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:50:44.938 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:50:44.938 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj47 = obj75 ']' 2024-05-08T16:50:44.938 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj47 td/test-erasure-eio/CHECK 2024-05-08T16:50:44.966 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:50:44.967 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:50:44.967 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj48 = obj75 ']' 2024-05-08T16:50:44.967 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj48 td/test-erasure-eio/CHECK 2024-05-08T16:50:44.995 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:50:44.996 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:50:44.996 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj49 = obj75 ']' 2024-05-08T16:50:44.996 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj49 td/test-erasure-eio/CHECK 2024-05-08T16:50:45.024 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:50:45.025 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:50:45.025 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj50 = obj75 ']' 2024-05-08T16:50:45.025 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj50 td/test-erasure-eio/CHECK 2024-05-08T16:50:45.053 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:50:45.054 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:50:45.054 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj51 = obj75 ']' 2024-05-08T16:50:45.054 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj51 td/test-erasure-eio/CHECK 2024-05-08T16:50:45.082 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:50:45.083 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:50:45.083 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj52 = obj75 ']' 2024-05-08T16:50:45.084 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj52 td/test-erasure-eio/CHECK 2024-05-08T16:50:45.112 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:50:45.113 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:50:45.113 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj53 = obj75 ']' 2024-05-08T16:50:45.113 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj53 td/test-erasure-eio/CHECK 2024-05-08T16:50:45.140 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:50:45.141 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:50:45.141 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj54 = obj75 ']' 2024-05-08T16:50:45.141 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj54 td/test-erasure-eio/CHECK 2024-05-08T16:50:45.169 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:50:45.171 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:50:45.171 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj55 = obj75 ']' 2024-05-08T16:50:45.171 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj55 td/test-erasure-eio/CHECK 2024-05-08T16:50:45.198 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:50:45.199 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:50:45.199 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj56 = obj75 ']' 2024-05-08T16:50:45.199 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj56 td/test-erasure-eio/CHECK 2024-05-08T16:50:45.227 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:50:45.228 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:50:45.228 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj57 = obj75 ']' 2024-05-08T16:50:45.229 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj57 td/test-erasure-eio/CHECK 2024-05-08T16:50:45.257 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:50:45.258 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:50:45.258 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj58 = obj75 ']' 2024-05-08T16:50:45.258 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj58 td/test-erasure-eio/CHECK 2024-05-08T16:50:45.286 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:50:45.287 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:50:45.287 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj59 = obj75 ']' 2024-05-08T16:50:45.287 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj59 td/test-erasure-eio/CHECK 2024-05-08T16:50:45.315 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:50:45.316 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:50:45.316 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj60 = obj75 ']' 2024-05-08T16:50:45.316 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj60 td/test-erasure-eio/CHECK 2024-05-08T16:50:45.345 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:50:45.346 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:50:45.346 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj61 = obj75 ']' 2024-05-08T16:50:45.346 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj61 td/test-erasure-eio/CHECK 2024-05-08T16:50:45.374 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:50:45.375 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:50:45.375 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj62 = obj75 ']' 2024-05-08T16:50:45.375 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj62 td/test-erasure-eio/CHECK 2024-05-08T16:50:45.404 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:50:45.405 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:50:45.405 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj63 = obj75 ']' 2024-05-08T16:50:45.405 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj63 td/test-erasure-eio/CHECK 2024-05-08T16:50:45.433 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:50:45.434 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:50:45.434 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj64 = obj75 ']' 2024-05-08T16:50:45.434 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj64 td/test-erasure-eio/CHECK 2024-05-08T16:50:45.462 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:50:45.463 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:50:45.463 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj65 = obj75 ']' 2024-05-08T16:50:45.463 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj65 td/test-erasure-eio/CHECK 2024-05-08T16:50:45.491 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:50:45.492 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:50:45.492 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj66 = obj75 ']' 2024-05-08T16:50:45.492 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj66 td/test-erasure-eio/CHECK 2024-05-08T16:50:45.521 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:50:45.521 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:50:45.521 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj67 = obj75 ']' 2024-05-08T16:50:45.522 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj67 td/test-erasure-eio/CHECK 2024-05-08T16:50:45.550 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:50:45.551 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:50:45.551 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj68 = obj75 ']' 2024-05-08T16:50:45.551 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj68 td/test-erasure-eio/CHECK 2024-05-08T16:50:45.579 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:50:45.580 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:50:45.580 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj69 = obj75 ']' 2024-05-08T16:50:45.580 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj69 td/test-erasure-eio/CHECK 2024-05-08T16:50:45.609 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:50:45.610 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:50:45.610 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj70 = obj75 ']' 2024-05-08T16:50:45.610 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj70 td/test-erasure-eio/CHECK 2024-05-08T16:50:45.638 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:50:45.639 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:50:45.639 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj71 = obj75 ']' 2024-05-08T16:50:45.639 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj71 td/test-erasure-eio/CHECK 2024-05-08T16:50:45.667 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:50:45.668 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:50:45.668 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj72 = obj75 ']' 2024-05-08T16:50:45.668 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj72 td/test-erasure-eio/CHECK 2024-05-08T16:50:45.697 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:50:45.698 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:50:45.698 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj73 = obj75 ']' 2024-05-08T16:50:45.698 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj73 td/test-erasure-eio/CHECK 2024-05-08T16:50:45.725 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:50:45.726 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:50:45.726 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj74 = obj75 ']' 2024-05-08T16:50:45.727 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj74 td/test-erasure-eio/CHECK 2024-05-08T16:50:45.755 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:50:45.756 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:50:45.756 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj75 = obj75 ']' 2024-05-08T16:50:45.756 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:684: TEST_ec_recovery_unfound: rados -p pool-jerasure get obj75 td/test-erasure-eio/CHECK 2024-05-08T16:50:45.781 INFO:tasks.workunit.client.0.smithi172.stderr:error getting pool-jerasure/obj75: (2) No such file or directory 2024-05-08T16:50:45.783 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:50:45.784 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj76 = obj75 ']' 2024-05-08T16:50:45.784 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj76 td/test-erasure-eio/CHECK 2024-05-08T16:50:45.856 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:50:45.857 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:50:45.857 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj77 = obj75 ']' 2024-05-08T16:50:45.857 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj77 td/test-erasure-eio/CHECK 2024-05-08T16:50:45.885 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:50:45.886 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:50:45.886 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj78 = obj75 ']' 2024-05-08T16:50:45.886 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj78 td/test-erasure-eio/CHECK 2024-05-08T16:50:45.916 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:50:45.917 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:50:45.917 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj79 = obj75 ']' 2024-05-08T16:50:45.917 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj79 td/test-erasure-eio/CHECK 2024-05-08T16:50:45.945 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:50:45.946 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:50:45.946 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj80 = obj75 ']' 2024-05-08T16:50:45.946 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj80 td/test-erasure-eio/CHECK 2024-05-08T16:50:45.974 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:50:45.974 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:50:45.974 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj81 = obj75 ']' 2024-05-08T16:50:45.975 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj81 td/test-erasure-eio/CHECK 2024-05-08T16:50:46.003 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:50:46.004 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:50:46.004 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj82 = obj75 ']' 2024-05-08T16:50:46.004 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj82 td/test-erasure-eio/CHECK 2024-05-08T16:50:46.032 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:50:46.033 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:50:46.033 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj83 = obj75 ']' 2024-05-08T16:50:46.033 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj83 td/test-erasure-eio/CHECK 2024-05-08T16:50:46.061 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:50:46.062 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:50:46.062 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj84 = obj75 ']' 2024-05-08T16:50:46.062 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj84 td/test-erasure-eio/CHECK 2024-05-08T16:50:46.090 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:50:46.091 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:50:46.091 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj85 = obj75 ']' 2024-05-08T16:50:46.091 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj85 td/test-erasure-eio/CHECK 2024-05-08T16:50:46.119 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:50:46.120 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:50:46.120 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj86 = obj75 ']' 2024-05-08T16:50:46.120 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj86 td/test-erasure-eio/CHECK 2024-05-08T16:50:46.148 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:50:46.149 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:50:46.149 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj87 = obj75 ']' 2024-05-08T16:50:46.149 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj87 td/test-erasure-eio/CHECK 2024-05-08T16:50:46.177 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:50:46.178 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:50:46.179 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj88 = obj75 ']' 2024-05-08T16:50:46.179 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj88 td/test-erasure-eio/CHECK 2024-05-08T16:50:46.207 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:50:46.208 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:50:46.208 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj89 = obj75 ']' 2024-05-08T16:50:46.208 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj89 td/test-erasure-eio/CHECK 2024-05-08T16:50:46.273 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:50:46.273 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:50:46.273 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj90 = obj75 ']' 2024-05-08T16:50:46.274 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj90 td/test-erasure-eio/CHECK 2024-05-08T16:50:46.302 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:50:46.303 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:50:46.303 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj91 = obj75 ']' 2024-05-08T16:50:46.303 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj91 td/test-erasure-eio/CHECK 2024-05-08T16:50:46.356 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:50:46.357 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:50:46.357 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj92 = obj75 ']' 2024-05-08T16:50:46.357 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj92 td/test-erasure-eio/CHECK 2024-05-08T16:50:46.385 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:50:46.386 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:50:46.386 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj93 = obj75 ']' 2024-05-08T16:50:46.386 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj93 td/test-erasure-eio/CHECK 2024-05-08T16:50:46.415 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:50:46.415 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:50:46.416 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj94 = obj75 ']' 2024-05-08T16:50:46.416 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj94 td/test-erasure-eio/CHECK 2024-05-08T16:50:46.444 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:50:46.445 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:50:46.445 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj95 = obj75 ']' 2024-05-08T16:50:46.445 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj95 td/test-erasure-eio/CHECK 2024-05-08T16:50:46.473 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:50:46.474 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:50:46.474 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj96 = obj75 ']' 2024-05-08T16:50:46.474 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj96 td/test-erasure-eio/CHECK 2024-05-08T16:50:46.502 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:50:46.503 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:50:46.503 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj97 = obj75 ']' 2024-05-08T16:50:46.503 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj97 td/test-erasure-eio/CHECK 2024-05-08T16:50:46.531 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:50:46.532 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:50:46.532 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj98 = obj75 ']' 2024-05-08T16:50:46.532 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj98 td/test-erasure-eio/CHECK 2024-05-08T16:50:46.560 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:50:46.561 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:50:46.561 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj99 = obj75 ']' 2024-05-08T16:50:46.561 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj99 td/test-erasure-eio/CHECK 2024-05-08T16:50:46.589 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:50:46.590 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:680: TEST_ec_recovery_unfound: for i in $(seq 1 $lastobj) 2024-05-08T16:50:46.590 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:682: TEST_ec_recovery_unfound: '[' obj100 = obj75 ']' 2024-05-08T16:50:46.590 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:686: TEST_ec_recovery_unfound: rados --pool pool-jerasure get obj100 td/test-erasure-eio/CHECK 2024-05-08T16:50:46.618 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:687: TEST_ec_recovery_unfound: diff -q td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:50:46.619 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:691: TEST_ec_recovery_unfound: rm -f td/test-erasure-eio/ORIGINAL td/test-erasure-eio/CHECK 2024-05-08T16:50:46.620 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:693: TEST_ec_recovery_unfound: delete_erasure_coded_pool pool-jerasure 2024-05-08T16:50:46.620 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:84: delete_erasure_coded_pool: local poolname=pool-jerasure 2024-05-08T16:50:46.620 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:85: delete_erasure_coded_pool: ceph osd pool delete pool-jerasure pool-jerasure --yes-i-really-really-mean-it 2024-05-08T16:50:47.123 INFO:tasks.workunit.client.0.smithi172.stderr:pool 'pool-jerasure' does not exist 2024-05-08T16:50:47.139 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:86: delete_erasure_coded_pool: ceph osd erasure-code-profile rm myprofile 2024-05-08T16:50:48.005 INFO:tasks.workunit.client.0.smithi172.stderr:erasure-code-profile myprofile does not exist 2024-05-08T16:50:48.025 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:42: run: teardown td/test-erasure-eio 2024-05-08T16:50:48.025 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-eio 2024-05-08T16:50:48.026 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-05-08T16:50:48.026 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-eio KILL 2024-05-08T16:50:48.026 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-08T16:50:48.026 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-08T16:50:48.027 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-08T16:50:48.027 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-08T16:50:48.027 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-08T16:50:48.340 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-08T16:50:48.341 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-05-08T16:50:48.341 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-05-08T16:50:48.342 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-05-08T16:50:48.343 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-05-08T16:50:48.343 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-05-08T16:50:48.344 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-05-08T16:50:48.345 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-08T16:50:48.345 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-05-08T16:50:48.346 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-05-08T16:50:48.346 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-08T16:50:48.347 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-05-08T16:50:48.349 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-05-08T16:50:48.349 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-eio 2024-05-08T16:50:48.392 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-05-08T16:50:48.392 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T16:50:48.392 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81110 2024-05-08T16:50:48.392 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.81110 2024-05-08T16:50:48.395 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-05-08T16:50:48.395 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-05-08T16:50:48.395 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:32: run: for func in $funcs 2024-05-08T16:50:48.395 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:33: run: setup td/test-erasure-eio 2024-05-08T16:50:48.395 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/test-erasure-eio 2024-05-08T16:50:48.395 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/test-erasure-eio 2024-05-08T16:50:48.396 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-eio 2024-05-08T16:50:48.396 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-05-08T16:50:48.396 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-eio KILL 2024-05-08T16:50:48.396 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-08T16:50:48.396 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-08T16:50:48.396 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-08T16:50:48.396 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-08T16:50:48.396 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-08T16:50:48.398 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-08T16:50:48.399 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-05-08T16:50:48.400 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-05-08T16:50:48.400 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-05-08T16:50:48.401 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-05-08T16:50:48.401 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-05-08T16:50:48.402 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-05-08T16:50:48.403 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-08T16:50:48.403 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-05-08T16:50:48.403 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-05-08T16:50:48.403 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-08T16:50:48.404 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-05-08T16:50:48.405 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-05-08T16:50:48.405 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-eio 2024-05-08T16:50:48.407 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-05-08T16:50:48.407 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T16:50:48.407 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81110 2024-05-08T16:50:48.407 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.81110 2024-05-08T16:50:48.408 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-05-08T16:50:48.408 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-05-08T16:50:48.408 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/test-erasure-eio 2024-05-08T16:50:48.410 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-05-08T16:50:48.410 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T16:50:48.410 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81110 2024-05-08T16:50:48.410 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.81110 2024-05-08T16:50:48.411 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-05-08T16:50:48.413 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-05-08T16:50:48.413 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-05-08T16:50:48.413 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/test-erasure-eio 1' TERM HUP INT 2024-05-08T16:50:48.413 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:34: run: run_mon td/test-erasure-eio a 2024-05-08T16:50:48.413 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/test-erasure-eio 2024-05-08T16:50:48.413 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-05-08T16:50:48.413 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-05-08T16:50:48.413 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-05-08T16:50:48.413 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/test-erasure-eio/a 2024-05-08T16:50:48.414 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-eio/a --run-dir=td/test-erasure-eio 2024-05-08T16:50:48.829 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-05-08T16:50:48.829 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-08T16:50:48.829 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-08T16:50:48.830 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-08T16:50:48.830 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T16:50:48.830 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81110 2024-05-08T16:50:48.830 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.81110/$cluster-$name.asok' 2024-05-08T16:50:48.831 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-eio/a '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.81110/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-eio/log --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-05-08T16:50:49.066 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-05-08T16:50:49.068 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-05-08T16:50:49.068 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-05-08T16:50:49.068 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-05-08T16:50:49.068 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-05-08T16:50:49.068 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-05-08T16:50:49.069 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-05-08T16:50:49.069 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-05-08T16:50:49.069 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-05-08T16:50:49.069 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-08T16:50:49.069 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T16:50:49.069 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81110 2024-05-08T16:50:49.070 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.81110/ceph-mon.a.asok 2024-05-08T16:50:49.070 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-05-08T16:50:49.070 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.81110/ceph-mon.a.asok config get fsid 2024-05-08T16:50:49.188 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-05-08T16:50:49.188 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-05-08T16:50:49.188 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-05-08T16:50:49.188 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-05-08T16:50:49.189 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-05-08T16:50:49.189 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-05-08T16:50:49.190 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-05-08T16:50:49.190 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-05-08T16:50:49.190 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-08T16:50:49.190 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T16:50:49.190 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81110 2024-05-08T16:50:49.190 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.81110/ceph-mon.a.asok 2024-05-08T16:50:49.190 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-05-08T16:50:49.190 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.81110/ceph-mon.a.asok config get mon_host 2024-05-08T16:50:49.295 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:35: run: run_mgr td/test-erasure-eio x 2024-05-08T16:50:49.295 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/test-erasure-eio 2024-05-08T16:50:49.296 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-05-08T16:50:49.296 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-05-08T16:50:49.296 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-05-08T16:50:49.296 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/test-erasure-eio/x 2024-05-08T16:50:49.296 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-05-08T16:50:50.229 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-05-08T16:50:50.229 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-08T16:50:50.229 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-08T16:50:50.229 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-08T16:50:50.229 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T16:50:50.230 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81110 2024-05-08T16:50:50.230 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.81110/$cluster-$name.asok' 2024-05-08T16:50:50.231 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-05-08T16:50:50.233 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-eio/x '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.81110/$cluster-$name.asok' --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-05-08T16:50:50.264 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:36: run: create_pool rbd 4 2024-05-08T16:50:50.264 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-05-08T16:50:50.619 INFO:tasks.workunit.client.0.smithi172.stderr:pool 'rbd' already exists 2024-05-08T16:50:50.636 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-05-08T16:50:51.638 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: get_asok_path mon.a 2024-05-08T16:50:51.638 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-05-08T16:50:51.638 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-05-08T16:50:51.640 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-08T16:50:51.640 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T16:50:51.640 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81110 2024-05-08T16:50:51.640 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.81110/ceph-mon.a.asok 2024-05-08T16:50:51.640 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: CEPH_ARGS= 2024-05-08T16:50:51.640 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: ceph --admin-daemon /tmp/ceph-asok.81110/ceph-mon.a.asok log flush 2024-05-08T16:50:51.738 INFO:tasks.workunit.client.0.smithi172.stdout:{} 2024-05-08T16:50:51.751 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:40: run: grep 'load: jerasure.*lrc' td/test-erasure-eio/mon.a.log 2024-05-08T16:50:51.754 INFO:tasks.workunit.client.0.smithi172.stdout:2024-05-08T16:50:48.872+0000 7f1ef7540c80 0 load: jerasure load: lrc 2024-05-08T16:50:51.756 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:41: run: TEST_ec_single_recovery_error td/test-erasure-eio 2024-05-08T16:50:51.756 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:390: TEST_ec_single_recovery_error: local dir=td/test-erasure-eio 2024-05-08T16:50:51.756 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:391: TEST_ec_single_recovery_error: local objname=myobject 2024-05-08T16:50:51.756 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:393: TEST_ec_single_recovery_error: setup_osds 7 2024-05-08T16:50:51.757 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:47: setup_osds: local count=7 2024-05-08T16:50:51.757 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:48: setup_osds: shift 2024-05-08T16:50:51.757 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: expr 7 - 1 2024-05-08T16:50:51.764 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: seq 0 6 2024-05-08T16:50:51.764 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-05-08T16:50:51.764 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 0 2024-05-08T16:50:51.765 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-05-08T16:50:51.765 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-05-08T16:50:51.765 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-05-08T16:50:51.765 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-05-08T16:50:51.765 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/0 2024-05-08T16:50:51.766 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=4457b0f1-028b-482b-8ee3-7ee05861e6cc --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-05-08T16:50:51.766 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-08T16:50:51.766 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-05-08T16:50:51.766 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-08T16:50:51.766 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/0' 2024-05-08T16:50:51.767 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/0/journal' 2024-05-08T16:50:51.767 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-05-08T16:50:51.767 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-05-08T16:50:51.767 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-05-08T16:50:51.767 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-05-08T16:50:51.768 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-08T16:50:51.768 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-08T16:50:51.768 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-08T16:50:51.768 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T16:50:51.768 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81110 2024-05-08T16:50:51.768 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.81110/$cluster-$name.asok' 2024-05-08T16:50:51.769 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.81110/$cluster-$name.asok' 2024-05-08T16:50:51.769 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-05-08T16:50:51.769 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-05-08T16:50:51.769 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-05-08T16:50:51.769 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-05-08T16:50:51.770 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-05-08T16:50:51.770 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-08T16:50:51.770 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-08T16:50:51.770 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-08T16:50:51.770 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-08T16:50:51.771 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-05-08T16:50:51.771 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-05-08T16:50:51.771 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/0 2024-05-08T16:50:51.771 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-05-08T16:50:51.771 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=6b452e80-9700-4053-90dc-3c1e9490c1a6 2024-05-08T16:50:51.772 INFO:tasks.workunit.client.0.smithi172.stdout:add osd0 6b452e80-9700-4053-90dc-3c1e9490c1a6 2024-05-08T16:50:51.773 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 6b452e80-9700-4053-90dc-3c1e9490c1a6' 2024-05-08T16:50:51.773 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-05-08T16:50:51.785 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBrrTtmCucLLxAAzXbihc8yIn13IMRkZimrlA== 2024-05-08T16:50:51.785 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBrrTtmCucLLxAAzXbihc8yIn13IMRkZimrlA=="}' 2024-05-08T16:50:51.785 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 6b452e80-9700-4053-90dc-3c1e9490c1a6 -i td/test-erasure-eio/0/new.json 2024-05-08T16:50:52.120 INFO:tasks.workunit.client.0.smithi172.stdout:0 2024-05-08T16:50:52.136 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/0/new.json 2024-05-08T16:50:52.138 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=4457b0f1-028b-482b-8ee3-7ee05861e6cc --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81110/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBrrTtmCucLLxAAzXbihc8yIn13IMRkZimrlA== --osd-uuid 6b452e80-9700-4053-90dc-3c1e9490c1a6 2024-05-08T16:50:52.167 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:50:52.168+0000 7fca5fd877c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:50:52.197 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:50:52.196+0000 7fca5fd877c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:50:52.198 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:50:52.200+0000 7fca5fd877c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:50:52.199 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:50:52.200+0000 7fca5fd877c0 -1 bluestore(td/test-erasure-eio/0/block) _read_bdev_label failed to open td/test-erasure-eio/0/block: (2) No such file or directory 2024-05-08T16:50:52.199 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:50:52.200+0000 7fca5fd877c0 -1 bluestore(td/test-erasure-eio/0/block) _read_bdev_label failed to open td/test-erasure-eio/0/block: (2) No such file or directory 2024-05-08T16:50:52.199 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:50:52.200+0000 7fca5fd877c0 -1 bluestore(td/test-erasure-eio/0/block) _read_bdev_label failed to open td/test-erasure-eio/0/block: (2) No such file or directory 2024-05-08T16:50:52.236 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:50:52.236+0000 7fca5fd877c0 -1 bluestore(td/test-erasure-eio/0) _read_fsid unparsable uuid 2024-05-08T16:50:59.727 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/0/keyring 2024-05-08T16:50:59.727 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-05-08T16:50:59.728 INFO:tasks.workunit.client.0.smithi172.stdout:adding osd0 key to auth repository 2024-05-08T16:50:59.729 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-05-08T16:50:59.729 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-05-08T16:51:00.347 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-05-08T16:51:00.348 INFO:tasks.workunit.client.0.smithi172.stdout:start osd.0 2024-05-08T16:51:00.348 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-05-08T16:51:00.348 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-05-08T16:51:00.348 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=4457b0f1-028b-482b-8ee3-7ee05861e6cc --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81110/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-05-08T16:51:00.349 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-05-08T16:51:00.379 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:51:00.380+0000 7f16373617c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:51:00.410 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:51:00.412+0000 7f16373617c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:51:00.412 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:51:00.412+0000 7f16373617c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:51:00.729 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-05-08T16:51:00.730 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-08T16:51:00.730 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-05-08T16:51:00.730 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-08T16:51:00.730 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-08T16:51:00.730 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:51:00.730 INFO:tasks.workunit.client.0.smithi172.stdout:0 2024-05-08T16:51:00.731 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-08T16:51:00.731 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:51:00.731 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:51:00.972 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:51:00.972+0000 7f16373617c0 -1 Falling back to public interface 2024-05-08T16:51:01.164 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:51:02.166 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:51:02.166 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:51:02.166 INFO:tasks.workunit.client.0.smithi172.stdout:1 2024-05-08T16:51:02.167 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-08T16:51:02.167 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:51:02.167 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:51:02.589 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:51:03.591 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:51:03.591 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:51:03.591 INFO:tasks.workunit.client.0.smithi172.stdout:2 2024-05-08T16:51:03.592 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-08T16:51:03.592 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:51:03.592 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:51:03.607 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:51:03.608+0000 7f16373617c0 -1 osd.0 0 log_to_monitors true 2024-05-08T16:51:04.240 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:51:05.242 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:51:05.242 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:51:05.242 INFO:tasks.workunit.client.0.smithi172.stdout:3 2024-05-08T16:51:05.243 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-05-08T16:51:05.243 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:51:05.243 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:51:05.742 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:51:06.743 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:51:06.744 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:51:06.744 INFO:tasks.workunit.client.0.smithi172.stdout:4 2024-05-08T16:51:06.744 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-05-08T16:51:06.745 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:51:06.745 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:51:07.195 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:51:08.196 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:51:08.197 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:51:08.197 INFO:tasks.workunit.client.0.smithi172.stdout:5 2024-05-08T16:51:08.197 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-05-08T16:51:08.198 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:51:08.198 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:51:08.734 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:51:09.735 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:51:09.736 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:51:09.736 INFO:tasks.workunit.client.0.smithi172.stdout:6 2024-05-08T16:51:09.737 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-05-08T16:51:09.737 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:51:09.737 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:51:10.178 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:51:11.180 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:51:11.180 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:51:11.180 INFO:tasks.workunit.client.0.smithi172.stdout:7 2024-05-08T16:51:11.181 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-05-08T16:51:11.181 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:51:11.181 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:51:11.758 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:51:12.760 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:51:12.760 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:51:12.760 INFO:tasks.workunit.client.0.smithi172.stdout:8 2024-05-08T16:51:12.761 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-05-08T16:51:12.761 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:51:12.761 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:51:13.227 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:51:14.229 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:51:14.230 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:51:14.230 INFO:tasks.workunit.client.0.smithi172.stdout:9 2024-05-08T16:51:14.231 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-05-08T16:51:14.231 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:51:14.231 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:51:14.734 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:51:15.736 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:51:15.736 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:51:15.737 INFO:tasks.workunit.client.0.smithi172.stdout:10 2024-05-08T16:51:15.737 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-05-08T16:51:15.737 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:51:15.737 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:51:16.296 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:51:17.298 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:51:17.298 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:51:17.298 INFO:tasks.workunit.client.0.smithi172.stdout:11 2024-05-08T16:51:17.299 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-05-08T16:51:17.299 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:51:17.299 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:51:17.795 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:51:18.796 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:51:18.797 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:51:18.797 INFO:tasks.workunit.client.0.smithi172.stdout:12 2024-05-08T16:51:18.797 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-05-08T16:51:18.797 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:51:18.798 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:51:19.447 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:51:20.449 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:51:20.449 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:51:20.449 INFO:tasks.workunit.client.0.smithi172.stdout:13 2024-05-08T16:51:20.450 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-05-08T16:51:20.450 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:51:20.450 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:51:21.263 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:51:22.264 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:51:22.265 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:51:22.265 INFO:tasks.workunit.client.0.smithi172.stdout:14 2024-05-08T16:51:22.265 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-05-08T16:51:22.266 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:51:22.266 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:51:23.077 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:51:24.079 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:51:24.079 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:51:24.079 INFO:tasks.workunit.client.0.smithi172.stdout:15 2024-05-08T16:51:24.079 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-05-08T16:51:24.080 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:51:24.080 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:51:24.984 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:51:25.986 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:51:25.986 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:51:25.986 INFO:tasks.workunit.client.0.smithi172.stdout:16 2024-05-08T16:51:25.987 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-05-08T16:51:25.987 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:51:25.987 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:51:26.893 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:51:27.895 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:51:27.895 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:51:27.895 INFO:tasks.workunit.client.0.smithi172.stdout:17 2024-05-08T16:51:27.896 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-05-08T16:51:27.896 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:51:27.896 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:51:28.708 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:51:29.709 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:51:29.709 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:51:29.710 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-05-08T16:51:29.710 INFO:tasks.workunit.client.0.smithi172.stdout:18 2024-05-08T16:51:29.710 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:51:29.710 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:51:30.502 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:51:31.504 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:51:31.504 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:51:31.504 INFO:tasks.workunit.client.0.smithi172.stdout:19 2024-05-08T16:51:31.505 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-05-08T16:51:31.505 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:51:31.505 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:51:32.076 INFO:tasks.workunit.client.0.smithi172.stdout:osd.0 up in weight 1 up_from 6 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1989575211,v1:127.0.0.1:6803/1989575211] [v2:127.0.0.1:6804/1989575211,v1:127.0.0.1:6805/1989575211] exists,up 6b452e80-9700-4053-90dc-3c1e9490c1a6 2024-05-08T16:51:32.079 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-08T16:51:32.079 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-08T16:51:32.079 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-08T16:51:32.079 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-05-08T16:51:32.079 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 1 2024-05-08T16:51:32.079 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-05-08T16:51:32.079 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-05-08T16:51:32.079 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-05-08T16:51:32.080 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-05-08T16:51:32.080 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/1 2024-05-08T16:51:32.080 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=4457b0f1-028b-482b-8ee3-7ee05861e6cc --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-05-08T16:51:32.080 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-08T16:51:32.080 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-05-08T16:51:32.080 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-08T16:51:32.080 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/1' 2024-05-08T16:51:32.080 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/1/journal' 2024-05-08T16:51:32.080 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-05-08T16:51:32.080 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-05-08T16:51:32.080 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-05-08T16:51:32.080 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-05-08T16:51:32.080 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-08T16:51:32.080 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-08T16:51:32.081 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-08T16:51:32.081 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T16:51:32.081 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81110 2024-05-08T16:51:32.081 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.81110/$cluster-$name.asok' 2024-05-08T16:51:32.083 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.81110/$cluster-$name.asok' 2024-05-08T16:51:32.083 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-05-08T16:51:32.083 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-05-08T16:51:32.083 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-05-08T16:51:32.083 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-05-08T16:51:32.083 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-05-08T16:51:32.083 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-08T16:51:32.083 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-08T16:51:32.083 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-08T16:51:32.083 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-08T16:51:32.083 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-05-08T16:51:32.083 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-05-08T16:51:32.083 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/1 2024-05-08T16:51:32.083 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-05-08T16:51:32.084 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=48c9cfae-a2d2-4706-92be-57e3d3056c19 2024-05-08T16:51:32.084 INFO:tasks.workunit.client.0.smithi172.stdout:add osd1 48c9cfae-a2d2-4706-92be-57e3d3056c19 2024-05-08T16:51:32.084 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 48c9cfae-a2d2-4706-92be-57e3d3056c19' 2024-05-08T16:51:32.084 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-05-08T16:51:32.106 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCUrTtmwdCTBhAAMfrJOmKE8e3c8RPBQlN8gg== 2024-05-08T16:51:32.106 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCUrTtmwdCTBhAAMfrJOmKE8e3c8RPBQlN8gg=="}' 2024-05-08T16:51:32.106 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 48c9cfae-a2d2-4706-92be-57e3d3056c19 -i td/test-erasure-eio/1/new.json 2024-05-08T16:51:33.148 INFO:tasks.workunit.client.0.smithi172.stdout:1 2024-05-08T16:51:33.172 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/1/new.json 2024-05-08T16:51:33.173 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=4457b0f1-028b-482b-8ee3-7ee05861e6cc --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81110/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCUrTtmwdCTBhAAMfrJOmKE8e3c8RPBQlN8gg== --osd-uuid 48c9cfae-a2d2-4706-92be-57e3d3056c19 2024-05-08T16:51:33.208 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:51:33.211+0000 7f8d551907c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:51:33.230 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:51:33.235+0000 7f8d551907c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:51:33.231 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:51:33.235+0000 7f8d551907c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:51:33.231 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:51:33.235+0000 7f8d551907c0 -1 bluestore(td/test-erasure-eio/1/block) _read_bdev_label failed to open td/test-erasure-eio/1/block: (2) No such file or directory 2024-05-08T16:51:33.232 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:51:33.235+0000 7f8d551907c0 -1 bluestore(td/test-erasure-eio/1/block) _read_bdev_label failed to open td/test-erasure-eio/1/block: (2) No such file or directory 2024-05-08T16:51:33.232 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:51:33.235+0000 7f8d551907c0 -1 bluestore(td/test-erasure-eio/1/block) _read_bdev_label failed to open td/test-erasure-eio/1/block: (2) No such file or directory 2024-05-08T16:51:33.324 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:51:33.327+0000 7f8d551907c0 -1 bluestore(td/test-erasure-eio/1) _read_fsid unparsable uuid 2024-05-08T16:51:40.816 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/1/keyring 2024-05-08T16:51:40.816 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-05-08T16:51:40.818 INFO:tasks.workunit.client.0.smithi172.stdout:adding osd1 key to auth repository 2024-05-08T16:51:40.818 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-05-08T16:51:40.818 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-05-08T16:51:41.377 INFO:tasks.workunit.client.0.smithi172.stdout:start osd.1 2024-05-08T16:51:41.378 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-05-08T16:51:41.378 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-05-08T16:51:41.378 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=4457b0f1-028b-482b-8ee3-7ee05861e6cc --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81110/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-05-08T16:51:41.378 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-05-08T16:51:41.378 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-05-08T16:51:41.409 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:51:41.411+0000 7f1db4c4b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:51:41.411 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:51:41.415+0000 7f1db4c4b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:51:41.412 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:51:41.415+0000 7f1db4c4b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:51:41.775 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-05-08T16:51:41.775 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-08T16:51:41.776 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-05-08T16:51:41.776 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-08T16:51:41.776 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-08T16:51:41.776 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:51:41.776 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-08T16:51:41.776 INFO:tasks.workunit.client.0.smithi172.stdout:0 2024-05-08T16:51:41.776 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:51:41.777 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T16:51:41.952 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:51:41.955+0000 7f1db4c4b7c0 -1 Falling back to public interface 2024-05-08T16:51:42.184 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:51:43.186 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:51:43.186 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:51:43.186 INFO:tasks.workunit.client.0.smithi172.stdout:1 2024-05-08T16:51:43.187 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-08T16:51:43.187 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:51:43.187 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T16:51:43.597 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:51:44.599 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:51:44.599 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:51:44.599 INFO:tasks.workunit.client.0.smithi172.stdout:2 2024-05-08T16:51:44.600 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-08T16:51:44.600 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:51:44.600 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T16:51:44.676 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:51:44.679+0000 7f1db4c4b7c0 -1 osd.1 0 log_to_monitors true 2024-05-08T16:51:45.093 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:51:46.094 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:51:46.095 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:51:46.095 INFO:tasks.workunit.client.0.smithi172.stdout:3 2024-05-08T16:51:46.095 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-05-08T16:51:46.096 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:51:46.096 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T16:51:46.729 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:51:47.731 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:51:47.731 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:51:47.731 INFO:tasks.workunit.client.0.smithi172.stdout:4 2024-05-08T16:51:47.732 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-05-08T16:51:47.732 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:51:47.732 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T16:51:48.162 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:51:49.164 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:51:49.164 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:51:49.164 INFO:tasks.workunit.client.0.smithi172.stdout:5 2024-05-08T16:51:49.165 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-05-08T16:51:49.165 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:51:49.165 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T16:51:49.654 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:51:50.656 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:51:50.656 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:51:50.656 INFO:tasks.workunit.client.0.smithi172.stdout:6 2024-05-08T16:51:50.656 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-05-08T16:51:50.657 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:51:50.657 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T16:51:51.188 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:51:52.191 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:51:52.191 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:51:52.192 INFO:tasks.workunit.client.0.smithi172.stdout:7 2024-05-08T16:51:52.192 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-05-08T16:51:52.193 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:51:52.193 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T16:51:52.729 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:51:53.730 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:51:53.731 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:51:53.731 INFO:tasks.workunit.client.0.smithi172.stdout:8 2024-05-08T16:51:53.731 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-05-08T16:51:53.732 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:51:53.732 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T16:51:54.149 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:51:55.151 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:51:55.151 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:51:55.151 INFO:tasks.workunit.client.0.smithi172.stdout:9 2024-05-08T16:51:55.152 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-05-08T16:51:55.152 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:51:55.152 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T16:51:55.683 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:51:56.684 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:51:56.685 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:51:56.685 INFO:tasks.workunit.client.0.smithi172.stdout:10 2024-05-08T16:51:56.685 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-05-08T16:51:56.685 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:51:56.685 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T16:51:57.101 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:51:58.102 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:51:58.103 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:51:58.103 INFO:tasks.workunit.client.0.smithi172.stdout:11 2024-05-08T16:51:58.103 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-05-08T16:51:58.103 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:51:58.104 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T16:51:58.687 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:51:59.689 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:51:59.689 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:51:59.689 INFO:tasks.workunit.client.0.smithi172.stdout:12 2024-05-08T16:51:59.690 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-05-08T16:51:59.690 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:51:59.690 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T16:52:00.169 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:52:01.170 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:52:01.171 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:52:01.171 INFO:tasks.workunit.client.0.smithi172.stdout:13 2024-05-08T16:52:01.172 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-05-08T16:52:01.172 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:52:01.172 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T16:52:02.510 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:52:03.512 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:52:03.512 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:52:03.513 INFO:tasks.workunit.client.0.smithi172.stdout:14 2024-05-08T16:52:03.513 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-05-08T16:52:03.513 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:52:03.513 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T16:52:04.531 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:52:05.533 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:52:05.533 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:52:05.533 INFO:tasks.workunit.client.0.smithi172.stdout:15 2024-05-08T16:52:05.534 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-05-08T16:52:05.534 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:52:05.534 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T16:52:06.301 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:52:07.302 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:52:07.302 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:52:07.303 INFO:tasks.workunit.client.0.smithi172.stdout:16 2024-05-08T16:52:07.303 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-05-08T16:52:07.303 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:52:07.303 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T16:52:08.289 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:52:09.291 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:52:09.291 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:52:09.291 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-05-08T16:52:09.291 INFO:tasks.workunit.client.0.smithi172.stdout:17 2024-05-08T16:52:09.292 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:52:09.292 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T16:52:09.883 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:52:10.885 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:52:10.885 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:52:10.885 INFO:tasks.workunit.client.0.smithi172.stdout:18 2024-05-08T16:52:10.886 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-05-08T16:52:10.886 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:52:10.886 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T16:52:11.574 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:52:12.578 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:52:12.578 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:52:12.579 INFO:tasks.workunit.client.0.smithi172.stdout:19 2024-05-08T16:52:12.579 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-05-08T16:52:12.579 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:52:12.579 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T16:52:13.070 INFO:tasks.workunit.client.0.smithi172.stdout:osd.1 up in weight 1 up_from 12 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/3675518369,v1:127.0.0.1:6811/3675518369] [v2:127.0.0.1:6812/3675518369,v1:127.0.0.1:6813/3675518369] exists,up 48c9cfae-a2d2-4706-92be-57e3d3056c19 2024-05-08T16:52:13.073 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-08T16:52:13.073 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-08T16:52:13.073 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-08T16:52:13.073 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-05-08T16:52:13.073 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 2 2024-05-08T16:52:13.073 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-05-08T16:52:13.073 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-05-08T16:52:13.073 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-05-08T16:52:13.073 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-05-08T16:52:13.073 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/2 2024-05-08T16:52:13.073 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=4457b0f1-028b-482b-8ee3-7ee05861e6cc --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-05-08T16:52:13.073 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-08T16:52:13.074 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-05-08T16:52:13.074 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-08T16:52:13.074 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/2' 2024-05-08T16:52:13.074 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/2/journal' 2024-05-08T16:52:13.074 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-05-08T16:52:13.074 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-05-08T16:52:13.074 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-05-08T16:52:13.074 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-05-08T16:52:13.074 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-08T16:52:13.074 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-08T16:52:13.074 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-08T16:52:13.074 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T16:52:13.074 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81110 2024-05-08T16:52:13.075 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.81110/$cluster-$name.asok' 2024-05-08T16:52:13.075 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.81110/$cluster-$name.asok' 2024-05-08T16:52:13.075 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-05-08T16:52:13.075 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-05-08T16:52:13.075 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-05-08T16:52:13.075 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-05-08T16:52:13.075 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-05-08T16:52:13.075 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-08T16:52:13.075 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-08T16:52:13.075 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-08T16:52:13.075 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-08T16:52:13.075 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-05-08T16:52:13.075 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-05-08T16:52:13.075 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/2 2024-05-08T16:52:13.076 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-05-08T16:52:13.076 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=4a1f09ae-2856-434b-afcb-c037aa452aa7 2024-05-08T16:52:13.076 INFO:tasks.workunit.client.0.smithi172.stdout:add osd2 4a1f09ae-2856-434b-afcb-c037aa452aa7 2024-05-08T16:52:13.077 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 4a1f09ae-2856-434b-afcb-c037aa452aa7' 2024-05-08T16:52:13.077 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-05-08T16:52:13.095 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQC9rTtmbpvjBRAAk7pgthNfligko1pOKihmyw== 2024-05-08T16:52:13.095 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQC9rTtmbpvjBRAAk7pgthNfligko1pOKihmyw=="}' 2024-05-08T16:52:13.095 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 4a1f09ae-2856-434b-afcb-c037aa452aa7 -i td/test-erasure-eio/2/new.json 2024-05-08T16:52:14.348 INFO:tasks.workunit.client.0.smithi172.stdout:2 2024-05-08T16:52:14.366 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/2/new.json 2024-05-08T16:52:14.368 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=4457b0f1-028b-482b-8ee3-7ee05861e6cc --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81110/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQC9rTtmbpvjBRAAk7pgthNfligko1pOKihmyw== --osd-uuid 4a1f09ae-2856-434b-afcb-c037aa452aa7 2024-05-08T16:52:14.401 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:52:14.402+0000 7fe49b69a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:52:14.473 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:52:14.474+0000 7fe49b69a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:52:14.474 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:52:14.478+0000 7fe49b69a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:52:14.475 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:52:14.478+0000 7fe49b69a7c0 -1 bluestore(td/test-erasure-eio/2/block) _read_bdev_label failed to open td/test-erasure-eio/2/block: (2) No such file or directory 2024-05-08T16:52:14.475 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:52:14.478+0000 7fe49b69a7c0 -1 bluestore(td/test-erasure-eio/2/block) _read_bdev_label failed to open td/test-erasure-eio/2/block: (2) No such file or directory 2024-05-08T16:52:14.475 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:52:14.478+0000 7fe49b69a7c0 -1 bluestore(td/test-erasure-eio/2/block) _read_bdev_label failed to open td/test-erasure-eio/2/block: (2) No such file or directory 2024-05-08T16:52:14.620 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:52:14.622+0000 7fe49b69a7c0 -1 bluestore(td/test-erasure-eio/2) _read_fsid unparsable uuid 2024-05-08T16:52:22.370 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/2/keyring 2024-05-08T16:52:22.370 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-05-08T16:52:22.372 INFO:tasks.workunit.client.0.smithi172.stdout:adding osd2 key to auth repository 2024-05-08T16:52:22.373 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-05-08T16:52:22.373 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-05-08T16:52:22.937 INFO:tasks.workunit.client.0.smithi172.stdout:start osd.2 2024-05-08T16:52:22.938 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-05-08T16:52:22.938 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-05-08T16:52:22.938 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-05-08T16:52:22.938 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=4457b0f1-028b-482b-8ee3-7ee05861e6cc --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81110/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-05-08T16:52:22.939 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-05-08T16:52:22.971 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:52:22.974+0000 7f1460d417c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:52:22.973 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:52:22.974+0000 7f1460d417c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:52:22.974 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:52:22.978+0000 7f1460d417c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:52:23.372 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-05-08T16:52:23.372 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-08T16:52:23.372 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-05-08T16:52:23.372 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-08T16:52:23.372 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-08T16:52:23.373 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:52:23.373 INFO:tasks.workunit.client.0.smithi172.stdout:0 2024-05-08T16:52:23.373 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-08T16:52:23.373 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:52:23.374 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T16:52:23.539 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:52:23.542+0000 7f1460d417c0 -1 Falling back to public interface 2024-05-08T16:52:23.800 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:52:24.802 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:52:24.802 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:52:24.802 INFO:tasks.workunit.client.0.smithi172.stdout:1 2024-05-08T16:52:24.803 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-08T16:52:24.803 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:52:24.803 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T16:52:25.238 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:52:26.240 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:52:26.240 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:52:26.240 INFO:tasks.workunit.client.0.smithi172.stdout:2 2024-05-08T16:52:26.241 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-08T16:52:26.241 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:52:26.241 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T16:52:26.335 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:52:26.338+0000 7f1460d417c0 -1 osd.2 0 log_to_monitors true 2024-05-08T16:52:26.737 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:52:27.738 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:52:27.738 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:52:27.739 INFO:tasks.workunit.client.0.smithi172.stdout:3 2024-05-08T16:52:27.739 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-05-08T16:52:27.739 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:52:27.739 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T16:52:28.210 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:52:29.212 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:52:29.212 INFO:tasks.workunit.client.0.smithi172.stdout:4 2024-05-08T16:52:29.213 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:52:29.213 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-05-08T16:52:29.213 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:52:29.214 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T16:52:29.833 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:52:30.834 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:52:30.835 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:52:30.835 INFO:tasks.workunit.client.0.smithi172.stdout:5 2024-05-08T16:52:30.835 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-05-08T16:52:30.835 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:52:30.836 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T16:52:31.291 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:52:32.293 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:52:32.293 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:52:32.293 INFO:tasks.workunit.client.0.smithi172.stdout:6 2024-05-08T16:52:32.294 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-05-08T16:52:32.294 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:52:32.294 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T16:52:32.768 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:52:33.770 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:52:33.770 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:52:33.770 INFO:tasks.workunit.client.0.smithi172.stdout:7 2024-05-08T16:52:33.771 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-05-08T16:52:33.771 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:52:33.771 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T16:52:34.288 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:52:35.289 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:52:35.289 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:52:35.289 INFO:tasks.workunit.client.0.smithi172.stdout:8 2024-05-08T16:52:35.290 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-05-08T16:52:35.290 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:52:35.290 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T16:52:35.785 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:52:36.787 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:52:36.787 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:52:36.788 INFO:tasks.workunit.client.0.smithi172.stdout:9 2024-05-08T16:52:36.788 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-05-08T16:52:36.788 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:52:36.788 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T16:52:37.404 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:52:38.405 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:52:38.406 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:52:38.406 INFO:tasks.workunit.client.0.smithi172.stdout:10 2024-05-08T16:52:38.406 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-05-08T16:52:38.406 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:52:38.406 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T16:52:38.869 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:52:39.871 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:52:39.871 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:52:39.871 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-05-08T16:52:39.871 INFO:tasks.workunit.client.0.smithi172.stdout:11 2024-05-08T16:52:39.872 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:52:39.872 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T16:52:40.464 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:52:41.465 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:52:41.466 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:52:41.466 INFO:tasks.workunit.client.0.smithi172.stdout:12 2024-05-08T16:52:41.467 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-05-08T16:52:41.467 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:52:41.467 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T16:52:41.943 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:52:42.944 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:52:42.945 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:52:42.945 INFO:tasks.workunit.client.0.smithi172.stdout:13 2024-05-08T16:52:42.945 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-05-08T16:52:42.946 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:52:42.946 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T16:52:43.603 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:52:44.605 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:52:44.605 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:52:44.605 INFO:tasks.workunit.client.0.smithi172.stdout:14 2024-05-08T16:52:44.606 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-05-08T16:52:44.606 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:52:44.606 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T16:52:45.352 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:52:46.354 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:52:46.354 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:52:46.354 INFO:tasks.workunit.client.0.smithi172.stdout:15 2024-05-08T16:52:46.355 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-05-08T16:52:46.355 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:52:46.355 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T16:52:47.221 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:52:48.223 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:52:48.223 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:52:48.223 INFO:tasks.workunit.client.0.smithi172.stdout:16 2024-05-08T16:52:48.224 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-05-08T16:52:48.224 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:52:48.224 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T16:52:48.895 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:52:49.897 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:52:49.897 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:52:49.897 INFO:tasks.workunit.client.0.smithi172.stdout:17 2024-05-08T16:52:49.898 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-05-08T16:52:49.898 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:52:49.898 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T16:52:50.739 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:52:51.741 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:52:51.741 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:52:51.741 INFO:tasks.workunit.client.0.smithi172.stdout:18 2024-05-08T16:52:51.742 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-05-08T16:52:51.742 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:52:51.742 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T16:52:52.395 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:52:53.396 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:52:53.396 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:52:53.397 INFO:tasks.workunit.client.0.smithi172.stdout:19 2024-05-08T16:52:53.397 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-05-08T16:52:53.397 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:52:53.398 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T16:52:54.634 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:52:55.636 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:52:55.636 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:52:55.636 INFO:tasks.workunit.client.0.smithi172.stdout:20 2024-05-08T16:52:55.637 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-05-08T16:52:55.637 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:52:55.637 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T16:52:56.095 INFO:tasks.workunit.client.0.smithi172.stdout:osd.2 up in weight 1 up_from 18 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/3704440413,v1:127.0.0.1:6819/3704440413] [v2:127.0.0.1:6820/3704440413,v1:127.0.0.1:6821/3704440413] exists,up 4a1f09ae-2856-434b-afcb-c037aa452aa7 2024-05-08T16:52:56.099 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-08T16:52:56.099 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-08T16:52:56.099 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-08T16:52:56.099 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-05-08T16:52:56.099 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 3 2024-05-08T16:52:56.099 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-05-08T16:52:56.099 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-05-08T16:52:56.099 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2024-05-08T16:52:56.099 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-05-08T16:52:56.099 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/3 2024-05-08T16:52:56.099 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=4457b0f1-028b-482b-8ee3-7ee05861e6cc --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-05-08T16:52:56.099 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-08T16:52:56.099 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-05-08T16:52:56.100 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-08T16:52:56.100 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/3' 2024-05-08T16:52:56.100 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/3/journal' 2024-05-08T16:52:56.100 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-05-08T16:52:56.100 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-05-08T16:52:56.100 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-05-08T16:52:56.100 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-05-08T16:52:56.100 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-08T16:52:56.100 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-08T16:52:56.100 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-08T16:52:56.100 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T16:52:56.100 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81110 2024-05-08T16:52:56.100 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.81110/$cluster-$name.asok' 2024-05-08T16:52:56.100 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.81110/$cluster-$name.asok' 2024-05-08T16:52:56.100 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-05-08T16:52:56.101 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-05-08T16:52:56.101 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-05-08T16:52:56.101 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-05-08T16:52:56.101 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-05-08T16:52:56.101 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-08T16:52:56.101 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-08T16:52:56.101 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-08T16:52:56.101 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-08T16:52:56.101 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-05-08T16:52:56.101 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-05-08T16:52:56.101 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/3 2024-05-08T16:52:56.102 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-05-08T16:52:56.102 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=b93c74ad-d75c-417d-bf2f-de7b935c3679 2024-05-08T16:52:56.102 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 b93c74ad-d75c-417d-bf2f-de7b935c3679' 2024-05-08T16:52:56.102 INFO:tasks.workunit.client.0.smithi172.stdout:add osd3 b93c74ad-d75c-417d-bf2f-de7b935c3679 2024-05-08T16:52:56.103 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-05-08T16:52:56.124 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDorTtm786yBxAAI9vTw++HcyWdR4Wn8QPMzg== 2024-05-08T16:52:56.124 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDorTtm786yBxAAI9vTw++HcyWdR4Wn8QPMzg=="}' 2024-05-08T16:52:56.125 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new b93c74ad-d75c-417d-bf2f-de7b935c3679 -i td/test-erasure-eio/3/new.json 2024-05-08T16:52:57.304 INFO:tasks.workunit.client.0.smithi172.stdout:3 2024-05-08T16:52:57.327 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/3/new.json 2024-05-08T16:52:57.328 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=4457b0f1-028b-482b-8ee3-7ee05861e6cc --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81110/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDorTtm786yBxAAI9vTw++HcyWdR4Wn8QPMzg== --osd-uuid b93c74ad-d75c-417d-bf2f-de7b935c3679 2024-05-08T16:52:57.363 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:52:57.365+0000 7f3dff8c77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:52:57.377 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:52:57.377+0000 7f3dff8c77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:52:57.378 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:52:57.381+0000 7f3dff8c77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:52:57.379 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:52:57.381+0000 7f3dff8c77c0 -1 bluestore(td/test-erasure-eio/3/block) _read_bdev_label failed to open td/test-erasure-eio/3/block: (2) No such file or directory 2024-05-08T16:52:57.379 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:52:57.381+0000 7f3dff8c77c0 -1 bluestore(td/test-erasure-eio/3/block) _read_bdev_label failed to open td/test-erasure-eio/3/block: (2) No such file or directory 2024-05-08T16:52:57.379 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:52:57.381+0000 7f3dff8c77c0 -1 bluestore(td/test-erasure-eio/3/block) _read_bdev_label failed to open td/test-erasure-eio/3/block: (2) No such file or directory 2024-05-08T16:52:57.568 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:52:57.573+0000 7f3dff8c77c0 -1 bluestore(td/test-erasure-eio/3) _read_fsid unparsable uuid 2024-05-08T16:53:05.378 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/3/keyring 2024-05-08T16:53:05.378 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-05-08T16:53:05.380 INFO:tasks.workunit.client.0.smithi172.stdout:adding osd3 key to auth repository 2024-05-08T16:53:05.380 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2024-05-08T16:53:05.380 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-05-08T16:53:05.940 INFO:tasks.workunit.client.0.smithi172.stdout:start osd.3 2024-05-08T16:53:05.941 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2024-05-08T16:53:05.941 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=4457b0f1-028b-482b-8ee3-7ee05861e6cc --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81110/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-05-08T16:53:05.941 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-05-08T16:53:05.941 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-05-08T16:53:05.942 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-05-08T16:53:05.972 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:53:05.973+0000 7f53ea4937c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:53:05.974 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:53:05.977+0000 7f53ea4937c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:53:05.975 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:53:05.977+0000 7f53ea4937c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:53:06.340 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2024-05-08T16:53:06.340 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-08T16:53:06.340 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-05-08T16:53:06.340 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-08T16:53:06.340 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-08T16:53:06.340 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:53:06.341 INFO:tasks.workunit.client.0.smithi172.stdout:0 2024-05-08T16:53:06.341 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-08T16:53:06.341 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:53:06.341 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T16:53:06.535 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:53:06.537+0000 7f53ea4937c0 -1 Falling back to public interface 2024-05-08T16:53:06.796 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:53:07.797 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:53:07.797 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:53:07.798 INFO:tasks.workunit.client.0.smithi172.stdout:1 2024-05-08T16:53:07.798 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-08T16:53:07.798 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:53:07.798 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T16:53:08.202 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:53:09.204 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:53:09.204 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:53:09.204 INFO:tasks.workunit.client.0.smithi172.stdout:2 2024-05-08T16:53:09.205 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-08T16:53:09.205 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:53:09.205 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T16:53:09.387 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:53:09.389+0000 7f53ea4937c0 -1 osd.3 0 log_to_monitors true 2024-05-08T16:53:09.786 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:53:10.787 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:53:10.788 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:53:10.788 INFO:tasks.workunit.client.0.smithi172.stdout:3 2024-05-08T16:53:10.788 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-05-08T16:53:10.788 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:53:10.789 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T16:53:11.437 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:53:12.439 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:53:12.439 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:53:12.439 INFO:tasks.workunit.client.0.smithi172.stdout:4 2024-05-08T16:53:12.440 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-05-08T16:53:12.440 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:53:12.440 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T16:53:13.155 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:53:14.157 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:53:14.157 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:53:14.157 INFO:tasks.workunit.client.0.smithi172.stdout:5 2024-05-08T16:53:14.158 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-05-08T16:53:14.158 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:53:14.158 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T16:53:14.622 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:53:15.624 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:53:15.624 INFO:tasks.workunit.client.0.smithi172.stdout:6 2024-05-08T16:53:15.625 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:53:15.625 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-05-08T16:53:15.625 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:53:15.626 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T16:53:16.097 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:53:17.098 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:53:17.098 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:53:17.099 INFO:tasks.workunit.client.0.smithi172.stdout:7 2024-05-08T16:53:17.099 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-05-08T16:53:17.099 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:53:17.099 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T16:53:17.629 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:53:18.631 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:53:18.631 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:53:18.632 INFO:tasks.workunit.client.0.smithi172.stdout:8 2024-05-08T16:53:18.632 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-05-08T16:53:18.632 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:53:18.632 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T16:53:19.207 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:53:20.208 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:53:20.208 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:53:20.209 INFO:tasks.workunit.client.0.smithi172.stdout:9 2024-05-08T16:53:20.209 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-05-08T16:53:20.209 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:53:20.209 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T16:53:20.673 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:53:21.674 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:53:21.675 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:53:21.675 INFO:tasks.workunit.client.0.smithi172.stdout:10 2024-05-08T16:53:21.675 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-05-08T16:53:21.675 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:53:21.675 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T16:53:22.328 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:53:23.330 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:53:23.330 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:53:23.330 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-05-08T16:53:23.330 INFO:tasks.workunit.client.0.smithi172.stdout:11 2024-05-08T16:53:23.331 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:53:23.331 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T16:53:23.978 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:53:24.980 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:53:24.980 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:53:24.980 INFO:tasks.workunit.client.0.smithi172.stdout:12 2024-05-08T16:53:24.981 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-05-08T16:53:24.981 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:53:24.981 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T16:53:25.510 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:53:26.512 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:53:26.512 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:53:26.513 INFO:tasks.workunit.client.0.smithi172.stdout:13 2024-05-08T16:53:26.513 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-05-08T16:53:26.513 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:53:26.513 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T16:53:26.945 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:53:27.946 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:53:27.946 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:53:27.947 INFO:tasks.workunit.client.0.smithi172.stdout:14 2024-05-08T16:53:27.947 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-05-08T16:53:27.947 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:53:27.947 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T16:53:28.991 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:53:29.993 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:53:29.993 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:53:29.993 INFO:tasks.workunit.client.0.smithi172.stdout:15 2024-05-08T16:53:29.994 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-05-08T16:53:29.994 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:53:29.994 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T16:53:30.444 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:53:31.446 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:53:31.446 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:53:31.447 INFO:tasks.workunit.client.0.smithi172.stdout:16 2024-05-08T16:53:31.447 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-05-08T16:53:31.447 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:53:31.447 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T16:53:32.302 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:53:33.304 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:53:33.304 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:53:33.304 INFO:tasks.workunit.client.0.smithi172.stdout:17 2024-05-08T16:53:33.305 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-05-08T16:53:33.305 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:53:33.305 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T16:53:34.660 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:53:35.662 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:53:35.662 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:53:35.662 INFO:tasks.workunit.client.0.smithi172.stdout:18 2024-05-08T16:53:35.663 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-05-08T16:53:35.663 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:53:35.663 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T16:53:36.490 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:53:37.493 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:53:37.493 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:53:37.493 INFO:tasks.workunit.client.0.smithi172.stdout:19 2024-05-08T16:53:37.494 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-05-08T16:53:37.494 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:53:37.494 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T16:53:38.926 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:53:39.929 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:53:39.929 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:53:39.929 INFO:tasks.workunit.client.0.smithi172.stdout:20 2024-05-08T16:53:39.930 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-05-08T16:53:39.930 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:53:39.930 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T16:53:40.542 INFO:tasks.workunit.client.0.smithi172.stdout:osd.3 up in weight 1 up_from 27 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/2827476115,v1:127.0.0.1:6827/2827476115] [v2:127.0.0.1:6828/2827476115,v1:127.0.0.1:6829/2827476115] exists,up b93c74ad-d75c-417d-bf2f-de7b935c3679 2024-05-08T16:53:40.545 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-08T16:53:40.545 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-08T16:53:40.545 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-08T16:53:40.545 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-05-08T16:53:40.545 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 4 2024-05-08T16:53:40.545 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-05-08T16:53:40.545 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-05-08T16:53:40.545 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=4 2024-05-08T16:53:40.546 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-05-08T16:53:40.546 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/4 2024-05-08T16:53:40.546 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=4457b0f1-028b-482b-8ee3-7ee05861e6cc --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-05-08T16:53:40.546 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-08T16:53:40.546 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-05-08T16:53:40.546 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-08T16:53:40.546 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/4' 2024-05-08T16:53:40.546 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/4/journal' 2024-05-08T16:53:40.546 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-05-08T16:53:40.546 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-05-08T16:53:40.546 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-05-08T16:53:40.546 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-05-08T16:53:40.546 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-08T16:53:40.547 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-08T16:53:40.547 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-08T16:53:40.547 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T16:53:40.547 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81110 2024-05-08T16:53:40.548 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.81110/$cluster-$name.asok' 2024-05-08T16:53:40.548 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.81110/$cluster-$name.asok' 2024-05-08T16:53:40.549 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-05-08T16:53:40.549 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-05-08T16:53:40.549 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-05-08T16:53:40.549 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-05-08T16:53:40.549 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-05-08T16:53:40.549 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-08T16:53:40.549 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-08T16:53:40.549 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-08T16:53:40.549 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-08T16:53:40.549 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-05-08T16:53:40.549 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-05-08T16:53:40.549 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/4 2024-05-08T16:53:40.549 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-05-08T16:53:40.550 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=779df22e-9b70-4afd-9bb6-5c5cffc07459 2024-05-08T16:53:40.550 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd4 779df22e-9b70-4afd-9bb6-5c5cffc07459' 2024-05-08T16:53:40.550 INFO:tasks.workunit.client.0.smithi172.stdout:add osd4 779df22e-9b70-4afd-9bb6-5c5cffc07459 2024-05-08T16:53:40.550 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-05-08T16:53:40.573 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAUrjtmJsVtIhAANBZMCZUnZoxwN6aPN/CX1g== 2024-05-08T16:53:40.573 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAUrjtmJsVtIhAANBZMCZUnZoxwN6aPN/CX1g=="}' 2024-05-08T16:53:40.573 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 779df22e-9b70-4afd-9bb6-5c5cffc07459 -i td/test-erasure-eio/4/new.json 2024-05-08T16:53:42.024 INFO:tasks.workunit.client.0.smithi172.stdout:4 2024-05-08T16:53:42.047 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/4/new.json 2024-05-08T16:53:42.048 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 4 --fsid=4457b0f1-028b-482b-8ee3-7ee05861e6cc --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/4 --osd-journal=td/test-erasure-eio/4/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81110/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAUrjtmJsVtIhAANBZMCZUnZoxwN6aPN/CX1g== --osd-uuid 779df22e-9b70-4afd-9bb6-5c5cffc07459 2024-05-08T16:53:42.083 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:53:42.084+0000 7ffb914147c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:53:42.114 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:53:42.116+0000 7ffb914147c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:53:42.115 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:53:42.116+0000 7ffb914147c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:53:42.116 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:53:42.116+0000 7ffb914147c0 -1 bluestore(td/test-erasure-eio/4/block) _read_bdev_label failed to open td/test-erasure-eio/4/block: (2) No such file or directory 2024-05-08T16:53:42.116 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:53:42.116+0000 7ffb914147c0 -1 bluestore(td/test-erasure-eio/4/block) _read_bdev_label failed to open td/test-erasure-eio/4/block: (2) No such file or directory 2024-05-08T16:53:42.116 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:53:42.116+0000 7ffb914147c0 -1 bluestore(td/test-erasure-eio/4/block) _read_bdev_label failed to open td/test-erasure-eio/4/block: (2) No such file or directory 2024-05-08T16:53:42.238 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:53:42.240+0000 7ffb914147c0 -1 bluestore(td/test-erasure-eio/4) _read_fsid unparsable uuid 2024-05-08T16:53:49.854 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/4/keyring 2024-05-08T16:53:49.855 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-05-08T16:53:49.855 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd4 key to auth repository 2024-05-08T16:53:49.856 INFO:tasks.workunit.client.0.smithi172.stdout:adding osd4 key to auth repository 2024-05-08T16:53:49.856 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-05-08T16:53:50.388 INFO:tasks.workunit.client.0.smithi172.stdout:start osd.4 2024-05-08T16:53:50.389 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.4 2024-05-08T16:53:50.389 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 4 --fsid=4457b0f1-028b-482b-8ee3-7ee05861e6cc --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/4 --osd-journal=td/test-erasure-eio/4/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81110/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-05-08T16:53:50.389 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-05-08T16:53:50.390 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-05-08T16:53:50.390 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-05-08T16:53:50.420 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:53:50.420+0000 7fabab0f57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:53:50.422 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:53:50.424+0000 7fabab0f57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:53:50.423 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:53:50.424+0000 7fabab0f57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:53:50.803 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 4 2024-05-08T16:53:50.803 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-08T16:53:50.803 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=4 2024-05-08T16:53:50.803 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-08T16:53:50.803 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-08T16:53:50.803 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:53:50.804 INFO:tasks.workunit.client.0.smithi172.stdout:0 2024-05-08T16:53:50.804 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-08T16:53:50.804 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:53:50.804 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-08T16:53:50.986 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:53:50.988+0000 7fabab0f57c0 -1 Falling back to public interface 2024-05-08T16:53:51.208 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:53:52.210 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:53:52.210 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:53:52.210 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-08T16:53:52.210 INFO:tasks.workunit.client.0.smithi172.stdout:1 2024-05-08T16:53:52.211 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:53:52.211 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-08T16:53:52.632 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:53:53.633 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:53:53.634 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:53:53.634 INFO:tasks.workunit.client.0.smithi172.stdout:2 2024-05-08T16:53:53.634 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-08T16:53:53.635 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:53:53.635 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-08T16:53:53.853 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:53:53.852+0000 7fabab0f57c0 -1 osd.4 0 log_to_monitors true 2024-05-08T16:53:54.171 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:53:55.173 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:53:55.173 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:53:55.173 INFO:tasks.workunit.client.0.smithi172.stdout:3 2024-05-08T16:53:55.174 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-05-08T16:53:55.174 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:53:55.174 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-08T16:53:55.753 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:53:56.755 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:53:56.755 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:53:56.755 INFO:tasks.workunit.client.0.smithi172.stdout:4 2024-05-08T16:53:56.756 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-05-08T16:53:56.756 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:53:56.756 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-08T16:53:57.320 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:53:58.321 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:53:58.321 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:53:58.322 INFO:tasks.workunit.client.0.smithi172.stdout:5 2024-05-08T16:53:58.322 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-05-08T16:53:58.322 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:53:58.322 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-08T16:53:58.999 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:54:00.000 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:54:00.000 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:54:00.001 INFO:tasks.workunit.client.0.smithi172.stdout:6 2024-05-08T16:54:00.001 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-05-08T16:54:00.001 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:54:00.001 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-08T16:54:00.578 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:54:01.579 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:54:01.579 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:54:01.580 INFO:tasks.workunit.client.0.smithi172.stdout:7 2024-05-08T16:54:01.580 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-05-08T16:54:01.580 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:54:01.580 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-08T16:54:02.080 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:54:03.081 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:54:03.081 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:54:03.082 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-05-08T16:54:03.082 INFO:tasks.workunit.client.0.smithi172.stdout:8 2024-05-08T16:54:03.082 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:54:03.082 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-08T16:54:03.688 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:54:04.690 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:54:04.690 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:54:04.690 INFO:tasks.workunit.client.0.smithi172.stdout:9 2024-05-08T16:54:04.691 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-05-08T16:54:04.691 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:54:04.691 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-08T16:54:05.295 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:54:06.297 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:54:06.297 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:54:06.297 INFO:tasks.workunit.client.0.smithi172.stdout:10 2024-05-08T16:54:06.298 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-05-08T16:54:06.298 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:54:06.298 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-08T16:54:06.938 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:54:07.939 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:54:07.939 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:54:07.940 INFO:tasks.workunit.client.0.smithi172.stdout:11 2024-05-08T16:54:07.940 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-05-08T16:54:07.940 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:54:07.940 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-08T16:54:08.591 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:54:09.593 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:54:09.593 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:54:09.594 INFO:tasks.workunit.client.0.smithi172.stdout:12 2024-05-08T16:54:09.594 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-05-08T16:54:09.595 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:54:09.595 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-08T16:54:10.023 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:54:11.025 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:54:11.025 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:54:11.025 INFO:tasks.workunit.client.0.smithi172.stdout:13 2024-05-08T16:54:11.026 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-05-08T16:54:11.026 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:54:11.026 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-08T16:54:11.458 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:54:12.459 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:54:12.459 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:54:12.460 INFO:tasks.workunit.client.0.smithi172.stdout:14 2024-05-08T16:54:12.460 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-05-08T16:54:12.460 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:54:12.461 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-08T16:54:13.513 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:54:14.514 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:54:14.515 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:54:14.515 INFO:tasks.workunit.client.0.smithi172.stdout:15 2024-05-08T16:54:14.515 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-05-08T16:54:14.516 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:54:14.516 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-08T16:54:15.375 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:54:16.377 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:54:16.378 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:54:16.378 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-05-08T16:54:16.378 INFO:tasks.workunit.client.0.smithi172.stdout:16 2024-05-08T16:54:16.379 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:54:16.379 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-08T16:54:17.324 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:54:18.326 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:54:18.326 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:54:18.326 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-05-08T16:54:18.326 INFO:tasks.workunit.client.0.smithi172.stdout:17 2024-05-08T16:54:18.326 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:54:18.327 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-08T16:54:19.126 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:54:20.128 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:54:20.129 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:54:20.129 INFO:tasks.workunit.client.0.smithi172.stdout:18 2024-05-08T16:54:20.129 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-05-08T16:54:20.129 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:54:20.130 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-08T16:54:20.664 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:54:21.666 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:54:21.666 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:54:21.666 INFO:tasks.workunit.client.0.smithi172.stdout:19 2024-05-08T16:54:21.667 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-05-08T16:54:21.667 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:54:21.667 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-08T16:54:22.924 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:54:23.926 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:54:23.927 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:54:23.927 INFO:tasks.workunit.client.0.smithi172.stdout:20 2024-05-08T16:54:23.928 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-05-08T16:54:23.928 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:54:23.928 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-08T16:54:24.651 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:54:25.652 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:54:25.653 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:54:25.653 INFO:tasks.workunit.client.0.smithi172.stdout:21 2024-05-08T16:54:25.654 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-05-08T16:54:25.654 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:54:25.654 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-05-08T16:54:26.179 INFO:tasks.workunit.client.0.smithi172.stdout:osd.4 up in weight 1 up_from 38 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/553025903,v1:127.0.0.1:6835/553025903] [v2:127.0.0.1:6836/553025903,v1:127.0.0.1:6837/553025903] exists,up 779df22e-9b70-4afd-9bb6-5c5cffc07459 2024-05-08T16:54:26.184 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-08T16:54:26.184 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-08T16:54:26.185 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-08T16:54:26.185 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-05-08T16:54:26.185 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 5 2024-05-08T16:54:26.185 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-05-08T16:54:26.185 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-05-08T16:54:26.185 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=5 2024-05-08T16:54:26.185 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-05-08T16:54:26.186 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/5 2024-05-08T16:54:26.186 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=4457b0f1-028b-482b-8ee3-7ee05861e6cc --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-05-08T16:54:26.186 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-08T16:54:26.186 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-05-08T16:54:26.186 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-08T16:54:26.186 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/5' 2024-05-08T16:54:26.186 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/5/journal' 2024-05-08T16:54:26.186 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-05-08T16:54:26.186 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-05-08T16:54:26.186 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-05-08T16:54:26.186 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-05-08T16:54:26.187 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-08T16:54:26.188 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-08T16:54:26.188 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-08T16:54:26.188 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T16:54:26.188 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81110 2024-05-08T16:54:26.188 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.81110/$cluster-$name.asok' 2024-05-08T16:54:26.188 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.81110/$cluster-$name.asok' 2024-05-08T16:54:26.189 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-05-08T16:54:26.189 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-05-08T16:54:26.189 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-05-08T16:54:26.189 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-05-08T16:54:26.189 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-05-08T16:54:26.189 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-08T16:54:26.190 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-08T16:54:26.190 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-08T16:54:26.190 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-08T16:54:26.190 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-05-08T16:54:26.190 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-05-08T16:54:26.190 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/5 2024-05-08T16:54:26.197 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-05-08T16:54:26.198 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=37546c5a-0c4d-4a70-8cbe-ab7793980c21 2024-05-08T16:54:26.198 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd5 37546c5a-0c4d-4a70-8cbe-ab7793980c21' 2024-05-08T16:54:26.198 INFO:tasks.workunit.client.0.smithi172.stdout:add osd5 37546c5a-0c4d-4a70-8cbe-ab7793980c21 2024-05-08T16:54:26.198 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-05-08T16:54:26.207 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBCrjtm50mcDBAABtlecNEPwa1UtskhIEgE/g== 2024-05-08T16:54:26.207 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBCrjtm50mcDBAABtlecNEPwa1UtskhIEgE/g=="}' 2024-05-08T16:54:26.207 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 37546c5a-0c4d-4a70-8cbe-ab7793980c21 -i td/test-erasure-eio/5/new.json 2024-05-08T16:54:27.191 INFO:tasks.workunit.client.0.smithi172.stdout:5 2024-05-08T16:54:27.216 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/5/new.json 2024-05-08T16:54:27.218 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 5 --fsid=4457b0f1-028b-482b-8ee3-7ee05861e6cc --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/5 --osd-journal=td/test-erasure-eio/5/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81110/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBCrjtm50mcDBAABtlecNEPwa1UtskhIEgE/g== --osd-uuid 37546c5a-0c4d-4a70-8cbe-ab7793980c21 2024-05-08T16:54:27.255 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:54:27.259+0000 7fd63929f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:54:27.373 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:54:27.375+0000 7fd63929f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:54:27.374 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:54:27.379+0000 7fd63929f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:54:27.374 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:54:27.379+0000 7fd63929f7c0 -1 bluestore(td/test-erasure-eio/5/block) _read_bdev_label failed to open td/test-erasure-eio/5/block: (2) No such file or directory 2024-05-08T16:54:27.374 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:54:27.379+0000 7fd63929f7c0 -1 bluestore(td/test-erasure-eio/5/block) _read_bdev_label failed to open td/test-erasure-eio/5/block: (2) No such file or directory 2024-05-08T16:54:27.374 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:54:27.379+0000 7fd63929f7c0 -1 bluestore(td/test-erasure-eio/5/block) _read_bdev_label failed to open td/test-erasure-eio/5/block: (2) No such file or directory 2024-05-08T16:54:27.513 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:54:27.515+0000 7fd63929f7c0 -1 bluestore(td/test-erasure-eio/5) _read_fsid unparsable uuid 2024-05-08T16:54:35.410 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/5/keyring 2024-05-08T16:54:35.410 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-05-08T16:54:35.411 INFO:tasks.workunit.client.0.smithi172.stdout:adding osd5 key to auth repository 2024-05-08T16:54:35.412 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd5 key to auth repository 2024-05-08T16:54:35.412 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/5/keyring auth add osd.5 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-05-08T16:54:35.971 INFO:tasks.workunit.client.0.smithi172.stdout:start osd.5 2024-05-08T16:54:35.972 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.5 2024-05-08T16:54:35.972 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 5 --fsid=4457b0f1-028b-482b-8ee3-7ee05861e6cc --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/5 --osd-journal=td/test-erasure-eio/5/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81110/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-05-08T16:54:35.972 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-05-08T16:54:35.972 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-05-08T16:54:35.973 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-05-08T16:54:36.003 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:54:36.007+0000 7f159b6917c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:54:36.005 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:54:36.011+0000 7f159b6917c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:54:36.006 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:54:36.011+0000 7f159b6917c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:54:36.385 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 5 2024-05-08T16:54:36.385 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-08T16:54:36.385 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=5 2024-05-08T16:54:36.385 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-08T16:54:36.385 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-08T16:54:36.385 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:54:36.385 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-08T16:54:36.385 INFO:tasks.workunit.client.0.smithi172.stdout:0 2024-05-08T16:54:36.386 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:54:36.386 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-05-08T16:54:36.554 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:54:36.559+0000 7f159b6917c0 -1 Falling back to public interface 2024-05-08T16:54:36.786 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:54:37.788 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:54:37.789 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:54:37.789 INFO:tasks.workunit.client.0.smithi172.stdout:1 2024-05-08T16:54:37.789 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-08T16:54:37.789 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:54:37.790 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-05-08T16:54:38.207 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:54:39.149 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:54:39.151+0000 7f159b6917c0 -1 osd.5 0 log_to_monitors true 2024-05-08T16:54:39.208 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:54:39.208 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:54:39.208 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-08T16:54:39.208 INFO:tasks.workunit.client.0.smithi172.stdout:2 2024-05-08T16:54:39.209 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:54:39.209 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-05-08T16:54:39.732 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:54:40.734 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:54:40.734 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:54:40.734 INFO:tasks.workunit.client.0.smithi172.stdout:3 2024-05-08T16:54:40.735 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-05-08T16:54:40.735 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:54:40.735 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-05-08T16:54:41.396 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:54:42.398 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:54:42.398 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:54:42.398 INFO:tasks.workunit.client.0.smithi172.stdout:4 2024-05-08T16:54:42.399 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-05-08T16:54:42.399 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:54:42.399 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-05-08T16:54:43.216 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:54:44.217 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:54:44.218 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:54:44.218 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-05-08T16:54:44.218 INFO:tasks.workunit.client.0.smithi172.stdout:5 2024-05-08T16:54:44.219 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:54:44.219 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-05-08T16:54:44.724 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:54:45.726 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:54:45.726 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:54:45.726 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-05-08T16:54:45.726 INFO:tasks.workunit.client.0.smithi172.stdout:6 2024-05-08T16:54:45.727 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:54:45.727 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-05-08T16:54:46.222 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:54:47.223 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:54:47.224 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:54:47.224 INFO:tasks.workunit.client.0.smithi172.stdout:7 2024-05-08T16:54:47.224 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-05-08T16:54:47.225 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:54:47.225 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-05-08T16:54:47.962 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:54:48.963 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:54:48.964 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:54:48.964 INFO:tasks.workunit.client.0.smithi172.stdout:8 2024-05-08T16:54:48.965 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-05-08T16:54:48.965 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:54:48.965 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-05-08T16:54:49.455 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:54:50.457 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:54:50.457 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:54:50.457 INFO:tasks.workunit.client.0.smithi172.stdout:9 2024-05-08T16:54:50.458 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-05-08T16:54:50.458 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:54:50.458 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-05-08T16:54:50.897 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:54:51.898 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:54:51.899 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:54:51.899 INFO:tasks.workunit.client.0.smithi172.stdout:10 2024-05-08T16:54:51.900 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-05-08T16:54:51.900 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:54:51.900 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-05-08T16:54:52.497 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:54:53.499 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:54:53.499 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:54:53.499 INFO:tasks.workunit.client.0.smithi172.stdout:11 2024-05-08T16:54:53.500 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-05-08T16:54:53.500 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:54:53.500 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-05-08T16:54:54.000 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:54:55.002 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:54:55.002 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:54:55.002 INFO:tasks.workunit.client.0.smithi172.stdout:12 2024-05-08T16:54:55.003 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-05-08T16:54:55.003 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:54:55.003 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-05-08T16:54:55.538 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:54:56.540 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:54:56.540 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:54:56.540 INFO:tasks.workunit.client.0.smithi172.stdout:13 2024-05-08T16:54:56.541 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-05-08T16:54:56.541 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:54:56.541 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-05-08T16:54:56.988 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:54:57.990 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:54:57.990 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:54:57.990 INFO:tasks.workunit.client.0.smithi172.stdout:14 2024-05-08T16:54:57.991 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-05-08T16:54:57.991 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:54:57.991 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-05-08T16:54:58.686 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:54:59.688 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:54:59.688 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:54:59.688 INFO:tasks.workunit.client.0.smithi172.stdout:15 2024-05-08T16:54:59.689 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-05-08T16:54:59.689 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:54:59.689 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-05-08T16:55:01.136 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:55:02.137 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:55:02.137 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:55:02.138 INFO:tasks.workunit.client.0.smithi172.stdout:16 2024-05-08T16:55:02.138 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-05-08T16:55:02.138 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:55:02.138 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-05-08T16:55:03.080 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:55:04.081 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:55:04.082 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:55:04.082 INFO:tasks.workunit.client.0.smithi172.stdout:17 2024-05-08T16:55:04.082 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-05-08T16:55:04.083 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:55:04.083 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-05-08T16:55:04.852 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:55:05.854 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:55:05.854 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:55:05.855 INFO:tasks.workunit.client.0.smithi172.stdout:18 2024-05-08T16:55:05.855 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-05-08T16:55:05.855 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:55:05.856 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-05-08T16:55:06.675 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:55:07.678 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:55:07.678 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:55:07.678 INFO:tasks.workunit.client.0.smithi172.stdout:19 2024-05-08T16:55:07.679 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-05-08T16:55:07.679 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:55:07.679 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-05-08T16:55:08.407 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:55:09.409 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:55:09.409 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:55:09.409 INFO:tasks.workunit.client.0.smithi172.stdout:20 2024-05-08T16:55:09.410 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-05-08T16:55:09.410 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:55:09.410 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-05-08T16:55:09.822 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:55:10.823 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:55:10.823 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:55:10.824 INFO:tasks.workunit.client.0.smithi172.stdout:21 2024-05-08T16:55:10.824 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-05-08T16:55:10.824 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:55:10.824 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-05-08T16:55:11.395 INFO:tasks.workunit.client.0.smithi172.stdout:osd.5 up in weight 1 up_from 48 up_thru 49 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6842/3288815710,v1:127.0.0.1:6843/3288815710] [v2:127.0.0.1:6844/3288815710,v1:127.0.0.1:6845/3288815710] exists,up 37546c5a-0c4d-4a70-8cbe-ab7793980c21 2024-05-08T16:55:11.399 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-08T16:55:11.399 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-08T16:55:11.399 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-08T16:55:11.399 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-05-08T16:55:11.399 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 6 2024-05-08T16:55:11.399 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-05-08T16:55:11.399 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-05-08T16:55:11.399 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=6 2024-05-08T16:55:11.399 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-05-08T16:55:11.399 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/6 2024-05-08T16:55:11.400 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=4457b0f1-028b-482b-8ee3-7ee05861e6cc --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-05-08T16:55:11.400 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-08T16:55:11.400 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-05-08T16:55:11.400 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-08T16:55:11.400 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/6' 2024-05-08T16:55:11.400 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/6/journal' 2024-05-08T16:55:11.400 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-05-08T16:55:11.400 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-05-08T16:55:11.400 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-05-08T16:55:11.400 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-05-08T16:55:11.400 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-08T16:55:11.400 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-08T16:55:11.400 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-08T16:55:11.400 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T16:55:11.400 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81110 2024-05-08T16:55:11.401 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.81110/$cluster-$name.asok' 2024-05-08T16:55:11.401 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.81110/$cluster-$name.asok' 2024-05-08T16:55:11.401 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-05-08T16:55:11.401 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-05-08T16:55:11.401 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-05-08T16:55:11.401 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-05-08T16:55:11.401 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-05-08T16:55:11.401 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-08T16:55:11.401 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-08T16:55:11.401 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-08T16:55:11.401 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-08T16:55:11.401 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-05-08T16:55:11.401 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-05-08T16:55:11.401 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/6 2024-05-08T16:55:11.402 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-05-08T16:55:11.402 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=a1ea6da4-bd8d-450a-918a-e4471c720c02 2024-05-08T16:55:11.402 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd6 a1ea6da4-bd8d-450a-918a-e4471c720c02' 2024-05-08T16:55:11.403 INFO:tasks.workunit.client.0.smithi172.stdout:add osd6 a1ea6da4-bd8d-450a-918a-e4471c720c02 2024-05-08T16:55:11.403 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-05-08T16:55:11.425 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBvrjtmWAqhGRAAYRaPut2MLOZNS3O+wICPqw== 2024-05-08T16:55:11.425 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBvrjtmWAqhGRAAYRaPut2MLOZNS3O+wICPqw=="}' 2024-05-08T16:55:11.425 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new a1ea6da4-bd8d-450a-918a-e4471c720c02 -i td/test-erasure-eio/6/new.json 2024-05-08T16:55:12.363 INFO:tasks.workunit.client.0.smithi172.stdout:6 2024-05-08T16:55:12.385 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/6/new.json 2024-05-08T16:55:12.386 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 6 --fsid=4457b0f1-028b-482b-8ee3-7ee05861e6cc --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/6 --osd-journal=td/test-erasure-eio/6/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81110/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBvrjtmWAqhGRAAYRaPut2MLOZNS3O+wICPqw== --osd-uuid a1ea6da4-bd8d-450a-918a-e4471c720c02 2024-05-08T16:55:12.422 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:55:12.426+0000 7fec7611f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:55:12.425 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:55:12.426+0000 7fec7611f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:55:12.426 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:55:12.430+0000 7fec7611f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:55:12.426 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:55:12.430+0000 7fec7611f7c0 -1 bluestore(td/test-erasure-eio/6/block) _read_bdev_label failed to open td/test-erasure-eio/6/block: (2) No such file or directory 2024-05-08T16:55:12.426 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:55:12.430+0000 7fec7611f7c0 -1 bluestore(td/test-erasure-eio/6/block) _read_bdev_label failed to open td/test-erasure-eio/6/block: (2) No such file or directory 2024-05-08T16:55:12.426 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:55:12.430+0000 7fec7611f7c0 -1 bluestore(td/test-erasure-eio/6/block) _read_bdev_label failed to open td/test-erasure-eio/6/block: (2) No such file or directory 2024-05-08T16:55:12.517 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:55:12.522+0000 7fec7611f7c0 -1 bluestore(td/test-erasure-eio/6) _read_fsid unparsable uuid 2024-05-08T16:55:18.770 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/6/keyring 2024-05-08T16:55:18.770 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-05-08T16:55:18.771 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd6 key to auth repository 2024-05-08T16:55:18.771 INFO:tasks.workunit.client.0.smithi172.stdout:adding osd6 key to auth repository 2024-05-08T16:55:18.771 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/6/keyring auth add osd.6 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-05-08T16:55:19.339 INFO:tasks.workunit.client.0.smithi172.stdout:start osd.6 2024-05-08T16:55:19.340 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.6 2024-05-08T16:55:19.340 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 6 --fsid=4457b0f1-028b-482b-8ee3-7ee05861e6cc --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/6 --osd-journal=td/test-erasure-eio/6/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81110/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-05-08T16:55:19.340 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-05-08T16:55:19.340 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-05-08T16:55:19.340 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-05-08T16:55:19.366 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:55:19.370+0000 7fce9dabb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:55:19.368 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:55:19.370+0000 7fce9dabb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:55:19.370 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:55:19.374+0000 7fce9dabb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:55:19.759 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 6 2024-05-08T16:55:19.760 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-08T16:55:19.760 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=6 2024-05-08T16:55:19.760 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-08T16:55:19.760 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-08T16:55:19.760 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:55:19.760 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-08T16:55:19.760 INFO:tasks.workunit.client.0.smithi172.stdout:0 2024-05-08T16:55:19.761 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-05-08T16:55:19.761 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:55:19.938 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:55:19.942+0000 7fce9dabb7c0 -1 Falling back to public interface 2024-05-08T16:55:20.202 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:55:21.203 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:55:21.204 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:55:21.204 INFO:tasks.workunit.client.0.smithi172.stdout:1 2024-05-08T16:55:21.204 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-08T16:55:21.205 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:55:21.205 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-05-08T16:55:21.608 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:55:22.467 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:55:22.470+0000 7fce9dabb7c0 -1 osd.6 0 log_to_monitors true 2024-05-08T16:55:22.610 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:55:22.610 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:55:22.610 INFO:tasks.workunit.client.0.smithi172.stdout:2 2024-05-08T16:55:22.611 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-08T16:55:22.611 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:55:22.611 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-05-08T16:55:23.403 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:55:24.405 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:55:24.405 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:55:24.405 INFO:tasks.workunit.client.0.smithi172.stdout:3 2024-05-08T16:55:24.406 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-05-08T16:55:24.406 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:55:24.406 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-05-08T16:55:25.106 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:55:26.108 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:55:26.108 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:55:26.108 INFO:tasks.workunit.client.0.smithi172.stdout:4 2024-05-08T16:55:26.109 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-05-08T16:55:26.109 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:55:26.109 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-05-08T16:55:26.568 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:55:27.569 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:55:27.570 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:55:27.570 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-05-08T16:55:27.570 INFO:tasks.workunit.client.0.smithi172.stdout:5 2024-05-08T16:55:27.571 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:55:27.571 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-05-08T16:55:28.041 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:55:29.042 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:55:29.043 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:55:29.043 INFO:tasks.workunit.client.0.smithi172.stdout:6 2024-05-08T16:55:29.044 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-05-08T16:55:29.044 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:55:29.044 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-05-08T16:55:29.529 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:55:30.530 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:55:30.531 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:55:30.531 INFO:tasks.workunit.client.0.smithi172.stdout:7 2024-05-08T16:55:30.531 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-05-08T16:55:30.531 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:55:30.531 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-05-08T16:55:30.967 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:55:31.969 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:55:31.969 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:55:31.969 INFO:tasks.workunit.client.0.smithi172.stdout:8 2024-05-08T16:55:31.970 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-05-08T16:55:31.970 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:55:31.970 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-05-08T16:55:32.430 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:55:33.431 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:55:33.431 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:55:33.432 INFO:tasks.workunit.client.0.smithi172.stdout:9 2024-05-08T16:55:33.432 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-05-08T16:55:33.432 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:55:33.432 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-05-08T16:55:33.936 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:55:34.938 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:55:34.938 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:55:34.938 INFO:tasks.workunit.client.0.smithi172.stdout:10 2024-05-08T16:55:34.939 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-05-08T16:55:34.939 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:55:34.939 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-05-08T16:55:35.401 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:55:36.402 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:55:36.402 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:55:36.402 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-05-08T16:55:36.403 INFO:tasks.workunit.client.0.smithi172.stdout:11 2024-05-08T16:55:36.403 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:55:36.403 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-05-08T16:55:36.861 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:55:37.862 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:55:37.863 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:55:37.863 INFO:tasks.workunit.client.0.smithi172.stdout:12 2024-05-08T16:55:37.863 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-05-08T16:55:37.863 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:55:37.864 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-05-08T16:55:38.565 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:55:39.567 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:55:39.567 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:55:39.568 INFO:tasks.workunit.client.0.smithi172.stdout:13 2024-05-08T16:55:39.568 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-05-08T16:55:39.568 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:55:39.568 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-05-08T16:55:40.669 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:55:41.671 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:55:41.672 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:55:41.672 INFO:tasks.workunit.client.0.smithi172.stdout:14 2024-05-08T16:55:41.672 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-05-08T16:55:41.673 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:55:41.673 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-05-08T16:55:42.145 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:55:43.147 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:55:43.147 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:55:43.148 INFO:tasks.workunit.client.0.smithi172.stdout:15 2024-05-08T16:55:43.148 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-05-08T16:55:43.148 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:55:43.148 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-05-08T16:55:44.283 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:55:45.285 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:55:45.285 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:55:45.285 INFO:tasks.workunit.client.0.smithi172.stdout:16 2024-05-08T16:55:45.286 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-05-08T16:55:45.286 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:55:45.286 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-05-08T16:55:46.558 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:55:47.560 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:55:47.560 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:55:47.560 INFO:tasks.workunit.client.0.smithi172.stdout:17 2024-05-08T16:55:47.561 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-05-08T16:55:47.561 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:55:47.561 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-05-08T16:55:48.500 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:55:49.501 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:55:49.501 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:55:49.502 INFO:tasks.workunit.client.0.smithi172.stdout:18 2024-05-08T16:55:49.502 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-05-08T16:55:49.502 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:55:49.502 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.6 up' 2024-05-08T16:55:49.997 INFO:tasks.workunit.client.0.smithi172.stdout:osd.6 up in weight 1 up_from 56 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6850/1338672064,v1:127.0.0.1:6851/1338672064] [v2:127.0.0.1:6852/1338672064,v1:127.0.0.1:6853/1338672064] exists,up a1ea6da4-bd8d-450a-918a-e4471c720c02 2024-05-08T16:55:49.998 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-08T16:55:49.998 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-08T16:55:49.998 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-08T16:55:49.998 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: get_asok_path osd.0 2024-05-08T16:55:49.998 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-05-08T16:55:49.998 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-05-08T16:55:49.999 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-08T16:55:49.999 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T16:55:50.000 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81110 2024-05-08T16:55:50.000 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.81110/ceph-osd.0.asok 2024-05-08T16:55:50.000 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: CEPH_ARGS= 2024-05-08T16:55:50.000 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: ceph --admin-daemon /tmp/ceph-asok.81110/ceph-osd.0.asok log flush 2024-05-08T16:55:50.098 INFO:tasks.workunit.client.0.smithi172.stdout:{} 2024-05-08T16:55:50.109 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:56: setup_osds: grep 'load: jerasure.*lrc' td/test-erasure-eio/osd.0.log 2024-05-08T16:55:50.117 INFO:tasks.workunit.client.0.smithi172.stdout:2024-05-08T16:51:00.980+0000 7f16373617c0 0 load: jerasure load: lrc 2024-05-08T16:55:50.118 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:395: TEST_ec_single_recovery_error: local poolname=pool-jerasure 2024-05-08T16:55:50.119 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:396: TEST_ec_single_recovery_error: create_erasure_coded_pool pool-jerasure 3 2 2024-05-08T16:55:50.119 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:67: create_erasure_coded_pool: local poolname=pool-jerasure 2024-05-08T16:55:50.119 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:68: create_erasure_coded_pool: shift 2024-05-08T16:55:50.119 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:69: create_erasure_coded_pool: local k=3 2024-05-08T16:55:50.119 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:70: create_erasure_coded_pool: shift 2024-05-08T16:55:50.119 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:71: create_erasure_coded_pool: local m=2 2024-05-08T16:55:50.119 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:72: create_erasure_coded_pool: shift 2024-05-08T16:55:50.119 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:74: create_erasure_coded_pool: ceph osd erasure-code-profile set myprofile plugin=jerasure k=3 m=2 crush-failure-domain=osd 2024-05-08T16:55:51.397 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:78: create_erasure_coded_pool: create_pool pool-jerasure 1 1 erasure myprofile 2024-05-08T16:55:51.397 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create pool-jerasure 1 1 erasure myprofile 2024-05-08T16:55:52.464 INFO:tasks.workunit.client.0.smithi172.stderr:pool 'pool-jerasure' already exists 2024-05-08T16:55:52.483 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-05-08T16:55:53.485 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:80: create_erasure_coded_pool: wait_for_clean 2024-05-08T16:55:53.485 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-05-08T16:55:53.486 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-05-08T16:55:53.486 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-05-08T16:55:53.486 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-05-08T16:55:53.488 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-05-08T16:55:53.488 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-05-08T16:55:53.488 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-05-08T16:55:53.488 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-05-08T16:55:53.488 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-05-08T16:55:53.616 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-05-08T16:55:53.616 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-05-08T16:55:53.616 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-05-08T16:55:53.616 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-05-08T16:55:53.616 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-05-08T16:55:53.616 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-05-08T16:55:54.123 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-05-08T16:55:54.123 INFO:tasks.workunit.client.0.smithi172.stderr:1 2024-05-08T16:55:54.123 INFO:tasks.workunit.client.0.smithi172.stderr:2 2024-05-08T16:55:54.123 INFO:tasks.workunit.client.0.smithi172.stderr:3 2024-05-08T16:55:54.123 INFO:tasks.workunit.client.0.smithi172.stderr:4 2024-05-08T16:55:54.123 INFO:tasks.workunit.client.0.smithi172.stderr:5 2024-05-08T16:55:54.123 INFO:tasks.workunit.client.0.smithi172.stderr:6' 2024-05-08T16:55:54.123 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-05-08T16:55:54.123 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T16:55:54.124 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-05-08T16:55:54.598 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=25769803834 2024-05-08T16:55:54.598 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 25769803834 2024-05-08T16:55:54.598 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803834' 2024-05-08T16:55:54.599 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T16:55:54.599 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-05-08T16:55:54.749 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=51539607606 2024-05-08T16:55:54.749 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 51539607606 2024-05-08T16:55:54.749 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803834 1-51539607606' 2024-05-08T16:55:54.749 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T16:55:54.749 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-05-08T16:55:54.929 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=77309411372 2024-05-08T16:55:54.929 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 77309411372 2024-05-08T16:55:54.929 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803834 1-51539607606 2-77309411372' 2024-05-08T16:55:54.930 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T16:55:54.930 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-05-08T16:55:55.081 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=115964117024 2024-05-08T16:55:55.081 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 115964117024 2024-05-08T16:55:55.082 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803834 1-51539607606 2-77309411372 3-115964117024' 2024-05-08T16:55:55.082 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T16:55:55.082 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-05-08T16:55:55.226 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=163208757274 2024-05-08T16:55:55.226 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 163208757274 2024-05-08T16:55:55.226 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803834 1-51539607606 2-77309411372 3-115964117024 4-163208757274' 2024-05-08T16:55:55.226 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T16:55:55.226 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-05-08T16:55:55.376 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=206158430224 2024-05-08T16:55:55.376 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 206158430224 2024-05-08T16:55:55.376 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803834 1-51539607606 2-77309411372 3-115964117024 4-163208757274 5-206158430224' 2024-05-08T16:55:55.376 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T16:55:55.376 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-05-08T16:55:55.538 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=240518168589 2024-05-08T16:55:55.538 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 240518168589 2024-05-08T16:55:55.538 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803834 1-51539607606 2-77309411372 3-115964117024 4-163208757274 5-206158430224 6-240518168589' 2024-05-08T16:55:55.538 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-08T16:55:55.539 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-25769803834 2024-05-08T16:55:55.539 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-08T16:55:55.540 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-05-08T16:55:55.540 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-25769803834 2024-05-08T16:55:55.540 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-08T16:55:55.541 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=25769803834 2024-05-08T16:55:55.541 INFO:tasks.workunit.client.0.smithi172.stdout:waiting osd.0 seq 25769803834 2024-05-08T16:55:55.541 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 25769803834' 2024-05-08T16:55:55.542 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-05-08T16:55:55.933 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 25769803834 -lt 25769803834 2024-05-08T16:55:55.933 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-08T16:55:55.933 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-51539607606 2024-05-08T16:55:55.933 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-08T16:55:55.934 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-05-08T16:55:55.936 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-51539607606 2024-05-08T16:55:55.936 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-08T16:55:55.937 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=51539607606 2024-05-08T16:55:55.937 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 51539607606' 2024-05-08T16:55:55.937 INFO:tasks.workunit.client.0.smithi172.stdout:waiting osd.1 seq 51539607606 2024-05-08T16:55:55.937 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-05-08T16:55:56.372 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 51539607606 -lt 51539607606 2024-05-08T16:55:56.372 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-08T16:55:56.372 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-77309411372 2024-05-08T16:55:56.373 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-08T16:55:56.374 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-05-08T16:55:56.375 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-77309411372 2024-05-08T16:55:56.375 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-08T16:55:56.376 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=77309411372 2024-05-08T16:55:56.376 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 77309411372' 2024-05-08T16:55:56.377 INFO:tasks.workunit.client.0.smithi172.stdout:waiting osd.2 seq 77309411372 2024-05-08T16:55:56.377 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-05-08T16:55:56.760 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 77309411372 -lt 77309411372 2024-05-08T16:55:56.760 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-08T16:55:56.761 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-115964117024 2024-05-08T16:55:56.761 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-08T16:55:56.762 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-05-08T16:55:56.763 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-115964117024 2024-05-08T16:55:56.763 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-08T16:55:56.764 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=115964117024 2024-05-08T16:55:56.764 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 115964117024' 2024-05-08T16:55:56.764 INFO:tasks.workunit.client.0.smithi172.stdout:waiting osd.3 seq 115964117024 2024-05-08T16:55:56.765 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-05-08T16:55:57.249 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 115964117025 -lt 115964117024 2024-05-08T16:55:57.249 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-08T16:55:57.249 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-163208757274 2024-05-08T16:55:57.249 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-08T16:55:57.250 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-05-08T16:55:57.251 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-163208757274 2024-05-08T16:55:57.251 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-08T16:55:57.252 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=163208757274 2024-05-08T16:55:57.253 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 163208757274' 2024-05-08T16:55:57.253 INFO:tasks.workunit.client.0.smithi172.stdout:waiting osd.4 seq 163208757274 2024-05-08T16:55:57.253 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-05-08T16:55:57.703 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 163208757275 -lt 163208757274 2024-05-08T16:55:57.703 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-08T16:55:57.704 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-206158430224 2024-05-08T16:55:57.704 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-08T16:55:57.706 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-05-08T16:55:57.707 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-206158430224 2024-05-08T16:55:57.707 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-08T16:55:57.708 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=206158430224 2024-05-08T16:55:57.708 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 206158430224' 2024-05-08T16:55:57.708 INFO:tasks.workunit.client.0.smithi172.stdout:waiting osd.5 seq 206158430224 2024-05-08T16:55:57.708 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-05-08T16:55:58.152 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 206158430224 -lt 206158430224 2024-05-08T16:55:58.152 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-08T16:55:58.153 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 6-240518168589 2024-05-08T16:55:58.153 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-08T16:55:58.154 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=6 2024-05-08T16:55:58.155 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-08T16:55:58.155 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 6-240518168589 2024-05-08T16:55:58.156 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=240518168589 2024-05-08T16:55:58.157 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.6 seq 240518168589' 2024-05-08T16:55:58.157 INFO:tasks.workunit.client.0.smithi172.stdout:waiting osd.6 seq 240518168589 2024-05-08T16:55:58.157 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 6 2024-05-08T16:55:58.542 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 240518168589 -lt 240518168589 2024-05-08T16:55:58.543 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-05-08T16:55:58.544 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-05-08T16:55:58.544 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-05-08T16:55:59.163 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-05-08T16:55:59.163 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-05-08T16:55:59.164 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-05-08T16:55:59.164 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-05-08T16:55:59.164 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-05-08T16:55:59.164 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-05-08T16:55:59.164 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-05-08T16:55:59.164 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-05-08T16:55:59.594 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-05-08T16:55:59.595 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-05-08T16:55:59.595 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-05-08T16:55:59.595 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-05-08T16:56:00.122 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-05-08T16:56:00.122 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-05-08T16:56:00.123 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-05-08T16:56:00.123 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:398: TEST_ec_single_recovery_error: rados_put td/test-erasure-eio pool-jerasure myobject 2024-05-08T16:56:00.123 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:90: rados_put: local dir=td/test-erasure-eio 2024-05-08T16:56:00.123 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:91: rados_put: local poolname=pool-jerasure 2024-05-08T16:56:00.123 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:92: rados_put: local objname=myobject 2024-05-08T16:56:00.123 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-05-08T16:56:00.123 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 AAA 2024-05-08T16:56:00.123 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-05-08T16:56:00.123 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 BBB 2024-05-08T16:56:00.123 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-05-08T16:56:00.123 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 CCCC 2024-05-08T16:56:00.123 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-05-08T16:56:00.123 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 DDDD 2024-05-08T16:56:00.123 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:100: rados_put: rados --pool pool-jerasure put myobject td/test-erasure-eio/ORIGINAL 2024-05-08T16:56:00.358 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:399: TEST_ec_single_recovery_error: inject_eio ec data pool-jerasure myobject td/test-erasure-eio 0 2024-05-08T16:56:00.358 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2394: inject_eio: local pooltype=ec 2024-05-08T16:56:00.358 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2395: inject_eio: shift 2024-05-08T16:56:00.358 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2396: inject_eio: local which=data 2024-05-08T16:56:00.358 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2397: inject_eio: shift 2024-05-08T16:56:00.359 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2398: inject_eio: local poolname=pool-jerasure 2024-05-08T16:56:00.359 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2399: inject_eio: shift 2024-05-08T16:56:00.359 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2400: inject_eio: local objname=myobject 2024-05-08T16:56:00.359 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2401: inject_eio: shift 2024-05-08T16:56:00.359 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2402: inject_eio: local dir=td/test-erasure-eio 2024-05-08T16:56:00.359 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2403: inject_eio: shift 2024-05-08T16:56:00.359 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2404: inject_eio: local shard_id=0 2024-05-08T16:56:00.359 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2405: inject_eio: shift 2024-05-08T16:56:00.359 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: get_osds pool-jerasure myobject 2024-05-08T16:56:00.359 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2024-05-08T16:56:00.359 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=myobject 2024-05-08T16:56:00.360 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure myobject 2024-05-08T16:56:00.360 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-05-08T16:56:00.805 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-05-08T16:56:00.805 INFO:tasks.workunit.client.0.smithi172.stderr:5 2024-05-08T16:56:00.806 INFO:tasks.workunit.client.0.smithi172.stderr:1 2024-05-08T16:56:00.806 INFO:tasks.workunit.client.0.smithi172.stderr:0 2024-05-08T16:56:00.806 INFO:tasks.workunit.client.0.smithi172.stderr:6' 2024-05-08T16:56:00.806 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 5 1 0 6 2024-05-08T16:56:00.806 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: initial_osds=('3' '5' '1' '0' '6') 2024-05-08T16:56:00.806 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: local -a initial_osds 2024-05-08T16:56:00.806 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2408: inject_eio: local osd_id=3 2024-05-08T16:56:00.806 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2409: inject_eio: '[' ec '!=' ec ']' 2024-05-08T16:56:00.807 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: cat td/test-erasure-eio/3/type 2024-05-08T16:56:00.807 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: type=bluestore 2024-05-08T16:56:00.807 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: set_config osd 3 bluestore_debug_inject_read_err true 2024-05-08T16:56:00.807 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-05-08T16:56:00.807 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-05-08T16:56:00.807 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-05-08T16:56:00.807 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-05-08T16:56:00.808 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-05-08T16:56:00.808 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-05-08T16:56:00.808 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-05-08T16:56:00.808 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-05-08T16:56:00.809 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-08T16:56:00.809 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T16:56:00.809 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81110 2024-05-08T16:56:00.809 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.81110/ceph-osd.3.asok 2024-05-08T16:56:00.809 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.81110/ceph-osd.3.asok config set bluestore_debug_inject_read_err true 2024-05-08T16:56:00.928 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-05-08T16:56:00.928 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local loop=0 2024-05-08T16:56:00.929 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: grep -q Invalid 2024-05-08T16:56:00.929 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: get_asok_path osd.3 2024-05-08T16:56:00.929 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-05-08T16:56:00.930 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-05-08T16:56:00.930 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-08T16:56:00.931 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T16:56:00.931 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81110 2024-05-08T16:56:00.931 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.81110/ceph-osd.3.asok 2024-05-08T16:56:00.931 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: CEPH_ARGS= 2024-05-08T16:56:00.931 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: ceph --admin-daemon /tmp/ceph-asok.81110/ceph-osd.3.asok injectdataerr pool-jerasure myobject 0 2024-05-08T16:56:01.050 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:401: TEST_ec_single_recovery_error: get_osds pool-jerasure myobject 2024-05-08T16:56:01.050 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2024-05-08T16:56:01.050 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=myobject 2024-05-08T16:56:01.051 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-05-08T16:56:01.051 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure myobject 2024-05-08T16:56:01.495 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-05-08T16:56:01.495 INFO:tasks.workunit.client.0.smithi172.stderr:5 2024-05-08T16:56:01.495 INFO:tasks.workunit.client.0.smithi172.stderr:1 2024-05-08T16:56:01.495 INFO:tasks.workunit.client.0.smithi172.stderr:0 2024-05-08T16:56:01.495 INFO:tasks.workunit.client.0.smithi172.stderr:6' 2024-05-08T16:56:01.495 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 5 1 0 6 2024-05-08T16:56:01.495 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:401: TEST_ec_single_recovery_error: initial_osds=('3' '5' '1' '0' '6') 2024-05-08T16:56:01.495 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:401: TEST_ec_single_recovery_error: local -a initial_osds 2024-05-08T16:56:01.495 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:402: TEST_ec_single_recovery_error: local last_osd=6 2024-05-08T16:56:01.495 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:404: TEST_ec_single_recovery_error: kill_daemons td/test-erasure-eio TERM osd.6 2024-05-08T16:56:01.496 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-08T16:56:01.496 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-08T16:56:01.496 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-08T16:56:01.496 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-08T16:56:01.496 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-08T16:56:01.806 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-08T16:56:01.806 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:405: TEST_ec_single_recovery_error: ceph osd down 6 2024-05-08T16:56:02.523 INFO:tasks.workunit.client.0.smithi172.stderr:osd.6 is already down. 2024-05-08T16:56:02.540 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:406: TEST_ec_single_recovery_error: ceph osd out 6 2024-05-08T16:56:03.445 INFO:tasks.workunit.client.0.smithi172.stderr:osd.6 is already out. 2024-05-08T16:56:03.462 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:409: TEST_ec_single_recovery_error: wait_for_clean 2024-05-08T16:56:03.462 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-05-08T16:56:03.462 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-05-08T16:56:03.463 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-05-08T16:56:03.463 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-05-08T16:56:03.463 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-05-08T16:56:03.464 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-05-08T16:56:03.464 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-05-08T16:56:03.464 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-05-08T16:56:03.464 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-05-08T16:56:03.582 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-05-08T16:56:03.582 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-05-08T16:56:03.582 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-05-08T16:56:03.582 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-05-08T16:56:03.582 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-05-08T16:56:03.582 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-05-08T16:56:04.207 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-05-08T16:56:04.207 INFO:tasks.workunit.client.0.smithi172.stderr:1 2024-05-08T16:56:04.208 INFO:tasks.workunit.client.0.smithi172.stderr:2 2024-05-08T16:56:04.208 INFO:tasks.workunit.client.0.smithi172.stderr:3 2024-05-08T16:56:04.208 INFO:tasks.workunit.client.0.smithi172.stderr:4 2024-05-08T16:56:04.208 INFO:tasks.workunit.client.0.smithi172.stderr:5 2024-05-08T16:56:04.208 INFO:tasks.workunit.client.0.smithi172.stderr:6' 2024-05-08T16:56:04.208 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-05-08T16:56:04.208 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T16:56:04.208 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-05-08T16:56:04.570 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=25769803838 2024-05-08T16:56:04.570 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 25769803838 2024-05-08T16:56:04.570 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803838' 2024-05-08T16:56:04.570 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T16:56:04.570 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-05-08T16:56:04.725 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=51539607610 2024-05-08T16:56:04.725 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 51539607610 2024-05-08T16:56:04.725 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803838 1-51539607610' 2024-05-08T16:56:04.725 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T16:56:04.725 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-05-08T16:56:04.985 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=77309411376 2024-05-08T16:56:04.985 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 77309411376 2024-05-08T16:56:04.985 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803838 1-51539607610 2-77309411376' 2024-05-08T16:56:04.985 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T16:56:04.985 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-05-08T16:56:05.137 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=115964117028 2024-05-08T16:56:05.138 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 115964117028 2024-05-08T16:56:05.138 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803838 1-51539607610 2-77309411376 3-115964117028' 2024-05-08T16:56:05.138 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T16:56:05.138 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-05-08T16:56:05.292 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=163208757278 2024-05-08T16:56:05.293 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 163208757278 2024-05-08T16:56:05.293 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803838 1-51539607610 2-77309411376 3-115964117028 4-163208757278' 2024-05-08T16:56:05.293 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T16:56:05.293 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-05-08T16:56:05.458 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=206158430228 2024-05-08T16:56:05.458 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 206158430228 2024-05-08T16:56:05.458 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803838 1-51539607610 2-77309411376 3-115964117028 4-163208757278 5-206158430228' 2024-05-08T16:56:05.459 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T16:56:05.459 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2024-05-08T16:56:05.610 INFO:tasks.workunit.client.0.smithi172.stderr:Error ENXIO: problem getting command descriptions from osd.6 2024-05-08T16:56:05.612 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq= 2024-05-08T16:56:05.612 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z '' 2024-05-08T16:56:05.612 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: continue 2024-05-08T16:56:05.612 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-08T16:56:05.613 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-25769803838 2024-05-08T16:56:05.613 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-08T16:56:05.614 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-05-08T16:56:05.615 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-25769803838 2024-05-08T16:56:05.615 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-08T16:56:05.616 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=25769803838 2024-05-08T16:56:05.616 INFO:tasks.workunit.client.0.smithi172.stdout:waiting osd.0 seq 25769803838 2024-05-08T16:56:05.616 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 25769803838' 2024-05-08T16:56:05.617 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-05-08T16:56:06.021 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 25769803838 -lt 25769803838 2024-05-08T16:56:06.021 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-08T16:56:06.022 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-51539607610 2024-05-08T16:56:06.022 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-08T16:56:06.024 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-05-08T16:56:06.025 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-51539607610 2024-05-08T16:56:06.025 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-08T16:56:06.026 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=51539607610 2024-05-08T16:56:06.027 INFO:tasks.workunit.client.0.smithi172.stdout:waiting osd.1 seq 51539607610 2024-05-08T16:56:06.027 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 51539607610' 2024-05-08T16:56:06.027 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-05-08T16:56:06.462 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 51539607610 -lt 51539607610 2024-05-08T16:56:06.462 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-08T16:56:06.463 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-77309411376 2024-05-08T16:56:06.463 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-08T16:56:06.465 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-05-08T16:56:06.466 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-77309411376 2024-05-08T16:56:06.466 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-08T16:56:06.467 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=77309411376 2024-05-08T16:56:06.467 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 77309411376' 2024-05-08T16:56:06.468 INFO:tasks.workunit.client.0.smithi172.stdout:waiting osd.2 seq 77309411376 2024-05-08T16:56:06.468 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-05-08T16:56:06.913 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 77309411374 -lt 77309411376 2024-05-08T16:56:06.913 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-05-08T16:56:07.914 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-05-08T16:56:07.915 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-05-08T16:56:08.331 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 77309411377 -lt 77309411376 2024-05-08T16:56:08.331 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-08T16:56:08.332 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-08T16:56:08.333 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-115964117028 2024-05-08T16:56:08.334 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-05-08T16:56:08.334 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-115964117028 2024-05-08T16:56:08.335 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-08T16:56:08.336 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=115964117028 2024-05-08T16:56:08.336 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 115964117028' 2024-05-08T16:56:08.336 INFO:tasks.workunit.client.0.smithi172.stdout:waiting osd.3 seq 115964117028 2024-05-08T16:56:08.337 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-05-08T16:56:08.825 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 115964117029 -lt 115964117028 2024-05-08T16:56:08.825 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-08T16:56:08.826 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 4-163208757278 2024-05-08T16:56:08.827 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-08T16:56:08.828 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=4 2024-05-08T16:56:08.829 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 4-163208757278 2024-05-08T16:56:08.829 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-08T16:56:08.830 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=163208757278 2024-05-08T16:56:08.830 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.4 seq 163208757278' 2024-05-08T16:56:08.831 INFO:tasks.workunit.client.0.smithi172.stdout:waiting osd.4 seq 163208757278 2024-05-08T16:56:08.831 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 4 2024-05-08T16:56:09.295 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 163208757279 -lt 163208757278 2024-05-08T16:56:09.295 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-08T16:56:09.296 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-08T16:56:09.296 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 5-206158430228 2024-05-08T16:56:09.297 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=5 2024-05-08T16:56:09.298 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 5-206158430228 2024-05-08T16:56:09.298 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-08T16:56:09.299 INFO:tasks.workunit.client.0.smithi172.stdout:waiting osd.5 seq 206158430228 2024-05-08T16:56:09.300 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=206158430228 2024-05-08T16:56:09.300 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.5 seq 206158430228' 2024-05-08T16:56:09.300 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 5 2024-05-08T16:56:09.714 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 206158430228 -lt 206158430228 2024-05-08T16:56:09.715 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-05-08T16:56:09.715 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-05-08T16:56:09.716 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-05-08T16:56:10.246 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-05-08T16:56:10.246 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-05-08T16:56:10.247 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-05-08T16:56:10.247 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-05-08T16:56:10.247 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-05-08T16:56:10.247 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-05-08T16:56:10.247 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-05-08T16:56:10.248 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-05-08T16:56:10.689 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-05-08T16:56:10.690 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-05-08T16:56:10.690 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-05-08T16:56:10.691 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-05-08T16:56:11.203 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 5 2024-05-08T16:56:11.203 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' -1 2024-05-08T16:56:11.203 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-05-08T16:56:11.203 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=4 2024-05-08T16:56:11.203 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-05-08T16:56:11.203 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-05-08T16:56:11.304 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-05-08T16:56:11.304 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-05-08T16:56:11.304 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-05-08T16:56:11.305 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-05-08T16:56:11.305 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-05-08T16:56:11.305 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-05-08T16:56:11.305 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-05-08T16:56:11.305 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-05-08T16:56:11.740 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-05-08T16:56:11.741 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-05-08T16:56:11.741 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-05-08T16:56:11.741 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-05-08T16:56:12.341 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-05-08T16:56:12.341 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-05-08T16:56:12.342 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-05-08T16:56:12.342 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:411: TEST_ec_single_recovery_error: rados_get td/test-erasure-eio pool-jerasure myobject 2024-05-08T16:56:12.342 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2024-05-08T16:56:12.342 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2024-05-08T16:56:12.342 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=myobject 2024-05-08T16:56:12.342 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=ok 2024-05-08T16:56:12.342 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' ok = fail ']' 2024-05-08T16:56:12.342 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:120: rados_get: rados --pool pool-jerasure get myobject td/test-erasure-eio/COPY 2024-05-08T16:56:12.447 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:121: rados_get: diff td/test-erasure-eio/ORIGINAL td/test-erasure-eio/COPY 2024-05-08T16:56:12.448 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:122: rados_get: rm td/test-erasure-eio/COPY 2024-05-08T16:56:12.449 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:413: TEST_ec_single_recovery_error: delete_erasure_coded_pool pool-jerasure 2024-05-08T16:56:12.449 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:84: delete_erasure_coded_pool: local poolname=pool-jerasure 2024-05-08T16:56:12.449 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:85: delete_erasure_coded_pool: ceph osd pool delete pool-jerasure pool-jerasure --yes-i-really-really-mean-it 2024-05-08T16:56:13.248 INFO:tasks.workunit.client.0.smithi172.stderr:pool 'pool-jerasure' does not exist 2024-05-08T16:56:13.265 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:86: delete_erasure_coded_pool: ceph osd erasure-code-profile rm myprofile 2024-05-08T16:56:13.910 INFO:tasks.workunit.client.0.smithi172.stderr:erasure-code-profile myprofile does not exist 2024-05-08T16:56:13.926 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:42: run: teardown td/test-erasure-eio 2024-05-08T16:56:13.926 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-eio 2024-05-08T16:56:13.926 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-05-08T16:56:13.927 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-eio KILL 2024-05-08T16:56:13.927 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-08T16:56:13.927 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-08T16:56:13.928 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-08T16:56:13.928 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-08T16:56:13.928 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-08T16:56:14.243 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-08T16:56:14.243 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-05-08T16:56:14.244 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-05-08T16:56:14.245 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-05-08T16:56:14.246 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-05-08T16:56:14.246 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-05-08T16:56:14.246 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-05-08T16:56:14.248 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-08T16:56:14.248 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-05-08T16:56:14.248 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-05-08T16:56:14.248 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-08T16:56:14.249 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-05-08T16:56:14.251 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-05-08T16:56:14.251 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-eio 2024-05-08T16:56:14.291 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-05-08T16:56:14.292 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T16:56:14.292 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81110 2024-05-08T16:56:14.292 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.81110 2024-05-08T16:56:14.293 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-05-08T16:56:14.294 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-05-08T16:56:14.294 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:32: run: for func in $funcs 2024-05-08T16:56:14.294 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:33: run: setup td/test-erasure-eio 2024-05-08T16:56:14.294 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/test-erasure-eio 2024-05-08T16:56:14.294 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/test-erasure-eio 2024-05-08T16:56:14.294 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-eio 2024-05-08T16:56:14.294 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-05-08T16:56:14.294 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-eio KILL 2024-05-08T16:56:14.294 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-08T16:56:14.294 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-08T16:56:14.294 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-08T16:56:14.294 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-08T16:56:14.294 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-08T16:56:14.297 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-08T16:56:14.297 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-05-08T16:56:14.298 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-05-08T16:56:14.298 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-05-08T16:56:14.300 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-05-08T16:56:14.300 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-05-08T16:56:14.300 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-05-08T16:56:14.301 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-08T16:56:14.301 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-05-08T16:56:14.302 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-05-08T16:56:14.302 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-08T16:56:14.302 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-05-08T16:56:14.303 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-05-08T16:56:14.303 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-eio 2024-05-08T16:56:14.305 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-05-08T16:56:14.305 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T16:56:14.305 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81110 2024-05-08T16:56:14.305 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.81110 2024-05-08T16:56:14.306 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-05-08T16:56:14.306 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-05-08T16:56:14.306 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/test-erasure-eio 2024-05-08T16:56:14.307 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-05-08T16:56:14.308 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T16:56:14.308 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81110 2024-05-08T16:56:14.308 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.81110 2024-05-08T16:56:14.310 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-05-08T16:56:14.310 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-05-08T16:56:14.310 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-05-08T16:56:14.310 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/test-erasure-eio 1' TERM HUP INT 2024-05-08T16:56:14.310 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:34: run: run_mon td/test-erasure-eio a 2024-05-08T16:56:14.310 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/test-erasure-eio 2024-05-08T16:56:14.310 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-05-08T16:56:14.311 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-05-08T16:56:14.311 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-05-08T16:56:14.311 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/test-erasure-eio/a 2024-05-08T16:56:14.311 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-eio/a --run-dir=td/test-erasure-eio 2024-05-08T16:56:14.721 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-05-08T16:56:14.721 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-08T16:56:14.721 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-08T16:56:14.722 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-08T16:56:14.722 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T16:56:14.722 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81110 2024-05-08T16:56:14.722 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.81110/$cluster-$name.asok' 2024-05-08T16:56:14.722 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-eio/a '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.81110/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-eio/log --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-05-08T16:56:14.941 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-05-08T16:56:14.943 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-05-08T16:56:14.943 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-05-08T16:56:14.943 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-05-08T16:56:14.943 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-05-08T16:56:14.944 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-05-08T16:56:14.944 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-05-08T16:56:14.944 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-05-08T16:56:14.944 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-05-08T16:56:14.944 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-08T16:56:14.945 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T16:56:14.945 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81110 2024-05-08T16:56:14.945 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.81110/ceph-mon.a.asok 2024-05-08T16:56:14.945 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-05-08T16:56:14.945 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.81110/ceph-mon.a.asok config get fsid 2024-05-08T16:56:15.062 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-05-08T16:56:15.063 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-05-08T16:56:15.063 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-05-08T16:56:15.063 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-05-08T16:56:15.063 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-05-08T16:56:15.063 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-05-08T16:56:15.063 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-05-08T16:56:15.064 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-05-08T16:56:15.064 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-08T16:56:15.064 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T16:56:15.064 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81110 2024-05-08T16:56:15.064 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.81110/ceph-mon.a.asok 2024-05-08T16:56:15.065 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-05-08T16:56:15.065 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.81110/ceph-mon.a.asok config get mon_host 2024-05-08T16:56:15.174 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:35: run: run_mgr td/test-erasure-eio x 2024-05-08T16:56:15.174 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/test-erasure-eio 2024-05-08T16:56:15.174 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-05-08T16:56:15.174 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-05-08T16:56:15.174 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-05-08T16:56:15.174 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/test-erasure-eio/x 2024-05-08T16:56:15.175 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-05-08T16:56:16.111 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-05-08T16:56:16.111 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-08T16:56:16.111 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-08T16:56:16.111 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-08T16:56:16.111 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T16:56:16.111 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81110 2024-05-08T16:56:16.112 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.81110/$cluster-$name.asok' 2024-05-08T16:56:16.112 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-05-08T16:56:16.113 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-eio/x '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.81110/$cluster-$name.asok' --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-05-08T16:56:16.145 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:36: run: create_pool rbd 4 2024-05-08T16:56:16.145 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-05-08T16:56:16.459 INFO:tasks.workunit.client.0.smithi172.stderr:pool 'rbd' already exists 2024-05-08T16:56:16.476 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-05-08T16:56:17.479 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: get_asok_path mon.a 2024-05-08T16:56:17.479 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-05-08T16:56:17.479 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-05-08T16:56:17.479 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-08T16:56:17.479 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T16:56:17.480 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81110 2024-05-08T16:56:17.480 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.81110/ceph-mon.a.asok 2024-05-08T16:56:17.480 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: CEPH_ARGS= 2024-05-08T16:56:17.480 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: ceph --admin-daemon /tmp/ceph-asok.81110/ceph-mon.a.asok log flush 2024-05-08T16:56:17.597 INFO:tasks.workunit.client.0.smithi172.stdout:{} 2024-05-08T16:56:17.610 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:40: run: grep 'load: jerasure.*lrc' td/test-erasure-eio/mon.a.log 2024-05-08T16:56:17.613 INFO:tasks.workunit.client.0.smithi172.stdout:2024-05-08T16:56:14.769+0000 7efc87fb7c80 0 load: jerasure load: lrc 2024-05-08T16:56:17.613 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:41: run: TEST_rados_get_bad_size_shard_0 td/test-erasure-eio 2024-05-08T16:56:17.614 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:302: TEST_rados_get_bad_size_shard_0: local dir=td/test-erasure-eio 2024-05-08T16:56:17.614 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:303: TEST_rados_get_bad_size_shard_0: setup_osds 4 2024-05-08T16:56:17.614 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:47: setup_osds: local count=4 2024-05-08T16:56:17.614 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:48: setup_osds: shift 2024-05-08T16:56:17.614 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: expr 4 - 1 2024-05-08T16:56:17.616 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: seq 0 3 2024-05-08T16:56:17.621 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-05-08T16:56:17.621 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 0 2024-05-08T16:56:17.621 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-05-08T16:56:17.622 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-05-08T16:56:17.622 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-05-08T16:56:17.622 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-05-08T16:56:17.622 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/0 2024-05-08T16:56:17.622 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=4457b0f1-028b-482b-8ee3-7ee05861e6cc --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-05-08T16:56:17.622 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-08T16:56:17.622 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-05-08T16:56:17.622 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-08T16:56:17.622 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/0' 2024-05-08T16:56:17.622 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/0/journal' 2024-05-08T16:56:17.622 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-05-08T16:56:17.622 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-05-08T16:56:17.622 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-05-08T16:56:17.623 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-05-08T16:56:17.623 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-08T16:56:17.623 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-08T16:56:17.623 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-08T16:56:17.623 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T16:56:17.623 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81110 2024-05-08T16:56:17.623 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.81110/$cluster-$name.asok' 2024-05-08T16:56:17.623 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.81110/$cluster-$name.asok' 2024-05-08T16:56:17.623 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-05-08T16:56:17.623 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-05-08T16:56:17.623 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-05-08T16:56:17.623 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-05-08T16:56:17.623 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-05-08T16:56:17.623 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-08T16:56:17.623 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-08T16:56:17.624 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-08T16:56:17.624 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-08T16:56:17.624 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-05-08T16:56:17.624 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-05-08T16:56:17.624 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/0 2024-05-08T16:56:17.624 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-05-08T16:56:17.624 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=7ea5f10d-bf66-4c7c-beb9-ecebdf6264f8 2024-05-08T16:56:17.625 INFO:tasks.workunit.client.0.smithi172.stdout:add osd0 7ea5f10d-bf66-4c7c-beb9-ecebdf6264f8 2024-05-08T16:56:17.625 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 7ea5f10d-bf66-4c7c-beb9-ecebdf6264f8' 2024-05-08T16:56:17.625 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-05-08T16:56:17.644 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCxrjtm0uS0JhAAgA7xVMQ4HO1iQjIfXZ2QAg== 2024-05-08T16:56:17.644 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCxrjtm0uS0JhAAgA7xVMQ4HO1iQjIfXZ2QAg=="}' 2024-05-08T16:56:17.644 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 7ea5f10d-bf66-4c7c-beb9-ecebdf6264f8 -i td/test-erasure-eio/0/new.json 2024-05-08T16:56:17.961 INFO:tasks.workunit.client.0.smithi172.stdout:0 2024-05-08T16:56:17.976 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/0/new.json 2024-05-08T16:56:17.977 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=4457b0f1-028b-482b-8ee3-7ee05861e6cc --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81110/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCxrjtm0uS0JhAAgA7xVMQ4HO1iQjIfXZ2QAg== --osd-uuid 7ea5f10d-bf66-4c7c-beb9-ecebdf6264f8 2024-05-08T16:56:18.005 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:56:18.009+0000 7f690c56b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:56:18.038 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:56:18.041+0000 7f690c56b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:56:18.039 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:56:18.041+0000 7f690c56b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:56:18.040 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:56:18.041+0000 7f690c56b7c0 -1 bluestore(td/test-erasure-eio/0/block) _read_bdev_label failed to open td/test-erasure-eio/0/block: (2) No such file or directory 2024-05-08T16:56:18.040 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:56:18.041+0000 7f690c56b7c0 -1 bluestore(td/test-erasure-eio/0/block) _read_bdev_label failed to open td/test-erasure-eio/0/block: (2) No such file or directory 2024-05-08T16:56:18.040 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:56:18.041+0000 7f690c56b7c0 -1 bluestore(td/test-erasure-eio/0/block) _read_bdev_label failed to open td/test-erasure-eio/0/block: (2) No such file or directory 2024-05-08T16:56:18.078 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:56:18.081+0000 7f690c56b7c0 -1 bluestore(td/test-erasure-eio/0) _read_fsid unparsable uuid 2024-05-08T16:56:24.959 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/0/keyring 2024-05-08T16:56:24.959 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-05-08T16:56:24.961 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-05-08T16:56:24.961 INFO:tasks.workunit.client.0.smithi172.stdout:adding osd0 key to auth repository 2024-05-08T16:56:24.961 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-05-08T16:56:25.526 INFO:tasks.workunit.client.0.smithi172.stdout:start osd.0 2024-05-08T16:56:25.527 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-05-08T16:56:25.527 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=4457b0f1-028b-482b-8ee3-7ee05861e6cc --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81110/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-05-08T16:56:25.527 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-05-08T16:56:25.527 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-05-08T16:56:25.527 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-05-08T16:56:25.557 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:56:25.560+0000 7f4f25b677c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:56:25.603 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:56:25.604+0000 7f4f25b677c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:56:25.604 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:56:25.604+0000 7f4f25b677c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:56:25.989 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-05-08T16:56:25.989 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-08T16:56:25.990 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-05-08T16:56:25.990 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-08T16:56:25.990 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-08T16:56:25.990 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:56:25.990 INFO:tasks.workunit.client.0.smithi172.stdout:0 2024-05-08T16:56:25.990 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-08T16:56:25.991 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:56:25.991 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:56:26.178 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:56:26.180+0000 7f4f25b677c0 -1 Falling back to public interface 2024-05-08T16:56:26.454 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:56:27.456 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:56:27.456 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:56:27.456 INFO:tasks.workunit.client.0.smithi172.stdout:1 2024-05-08T16:56:27.457 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-08T16:56:27.457 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:56:27.457 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:56:27.875 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:56:28.876 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:56:28.877 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:56:28.877 INFO:tasks.workunit.client.0.smithi172.stdout:2 2024-05-08T16:56:28.877 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-08T16:56:28.877 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:56:28.877 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:56:28.902 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:56:28.904+0000 7f4f25b677c0 -1 osd.0 0 log_to_monitors true 2024-05-08T16:56:29.451 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:56:30.453 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:56:30.453 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:56:30.453 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-05-08T16:56:30.453 INFO:tasks.workunit.client.0.smithi172.stdout:3 2024-05-08T16:56:30.454 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:56:30.454 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:56:30.875 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:56:31.876 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:56:31.877 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:56:31.877 INFO:tasks.workunit.client.0.smithi172.stdout:4 2024-05-08T16:56:31.877 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-05-08T16:56:31.877 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:56:31.878 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:56:32.303 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:56:33.305 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:56:33.305 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:56:33.305 INFO:tasks.workunit.client.0.smithi172.stdout:5 2024-05-08T16:56:33.306 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-05-08T16:56:33.306 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:56:33.306 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:56:33.800 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:56:34.802 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:56:34.802 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:56:34.802 INFO:tasks.workunit.client.0.smithi172.stdout:6 2024-05-08T16:56:34.803 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-05-08T16:56:34.803 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:56:34.803 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:56:35.263 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:56:36.265 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:56:36.265 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:56:36.266 INFO:tasks.workunit.client.0.smithi172.stdout:7 2024-05-08T16:56:36.266 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-05-08T16:56:36.266 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:56:36.266 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:56:36.826 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:56:37.828 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:56:37.828 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:56:37.828 INFO:tasks.workunit.client.0.smithi172.stdout:8 2024-05-08T16:56:37.829 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-05-08T16:56:37.829 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:56:37.829 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:56:38.278 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:56:39.280 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:56:39.281 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:56:39.281 INFO:tasks.workunit.client.0.smithi172.stdout:9 2024-05-08T16:56:39.281 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-05-08T16:56:39.281 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:56:39.282 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:56:39.790 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:56:40.792 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:56:40.792 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:56:40.792 INFO:tasks.workunit.client.0.smithi172.stdout:10 2024-05-08T16:56:40.793 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-05-08T16:56:40.793 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:56:40.793 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:56:41.308 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:56:42.309 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:56:42.309 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:56:42.309 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-05-08T16:56:42.310 INFO:tasks.workunit.client.0.smithi172.stdout:11 2024-05-08T16:56:42.310 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:56:42.310 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:56:42.835 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:56:43.836 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:56:43.836 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:56:43.836 INFO:tasks.workunit.client.0.smithi172.stdout:12 2024-05-08T16:56:43.837 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-05-08T16:56:43.837 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:56:43.837 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:56:44.787 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:56:45.789 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:56:45.789 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:56:45.789 INFO:tasks.workunit.client.0.smithi172.stdout:13 2024-05-08T16:56:45.790 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-05-08T16:56:45.790 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:56:45.790 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:56:46.827 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:56:47.828 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:56:47.829 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:56:47.829 INFO:tasks.workunit.client.0.smithi172.stdout:14 2024-05-08T16:56:47.829 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-05-08T16:56:47.829 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:56:47.829 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:56:48.654 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:56:49.656 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:56:49.656 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:56:49.656 INFO:tasks.workunit.client.0.smithi172.stdout:15 2024-05-08T16:56:49.657 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-05-08T16:56:49.657 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:56:49.657 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:56:51.148 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:56:52.150 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:56:52.150 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:56:52.150 INFO:tasks.workunit.client.0.smithi172.stdout:16 2024-05-08T16:56:52.151 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-05-08T16:56:52.151 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:56:52.151 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:56:53.434 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:56:54.436 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:56:54.436 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:56:54.436 INFO:tasks.workunit.client.0.smithi172.stdout:17 2024-05-08T16:56:54.437 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-05-08T16:56:54.437 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:56:54.437 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T16:56:54.941 INFO:tasks.workunit.client.0.smithi172.stdout:osd.0 up in weight 1 up_from 6 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/115373454,v1:127.0.0.1:6803/115373454] [v2:127.0.0.1:6804/115373454,v1:127.0.0.1:6805/115373454] exists,up 7ea5f10d-bf66-4c7c-beb9-ecebdf6264f8 2024-05-08T16:56:54.943 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-08T16:56:54.943 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-08T16:56:54.943 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-08T16:56:54.943 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-05-08T16:56:54.943 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 1 2024-05-08T16:56:54.943 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-05-08T16:56:54.943 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-05-08T16:56:54.943 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-05-08T16:56:54.944 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-05-08T16:56:54.944 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/1 2024-05-08T16:56:54.944 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=4457b0f1-028b-482b-8ee3-7ee05861e6cc --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-05-08T16:56:54.944 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-08T16:56:54.944 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-05-08T16:56:54.944 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-08T16:56:54.944 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/1' 2024-05-08T16:56:54.944 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/1/journal' 2024-05-08T16:56:54.944 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-05-08T16:56:54.944 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-05-08T16:56:54.944 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-05-08T16:56:54.944 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-05-08T16:56:54.944 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-08T16:56:54.944 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-08T16:56:54.946 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-08T16:56:54.946 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T16:56:54.947 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81110 2024-05-08T16:56:54.947 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.81110/$cluster-$name.asok' 2024-05-08T16:56:54.947 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.81110/$cluster-$name.asok' 2024-05-08T16:56:54.947 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-05-08T16:56:54.947 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-05-08T16:56:54.947 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-05-08T16:56:54.947 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-05-08T16:56:54.947 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-05-08T16:56:54.947 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-08T16:56:54.947 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-08T16:56:54.947 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-08T16:56:54.947 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-08T16:56:54.947 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-05-08T16:56:54.947 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-05-08T16:56:54.948 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/1 2024-05-08T16:56:54.948 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-05-08T16:56:54.948 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=0f51dcbd-8561-4314-b628-17fe241f145f 2024-05-08T16:56:54.948 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 0f51dcbd-8561-4314-b628-17fe241f145f' 2024-05-08T16:56:54.949 INFO:tasks.workunit.client.0.smithi172.stdout:add osd1 0f51dcbd-8561-4314-b628-17fe241f145f 2024-05-08T16:56:54.949 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-05-08T16:56:54.965 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDWrjtmqWLcORAAd8YQ32KdfmLYgoPNFGE3rw== 2024-05-08T16:56:54.965 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDWrjtmqWLcORAAd8YQ32KdfmLYgoPNFGE3rw=="}' 2024-05-08T16:56:54.965 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 0f51dcbd-8561-4314-b628-17fe241f145f -i td/test-erasure-eio/1/new.json 2024-05-08T16:56:55.889 INFO:tasks.workunit.client.0.smithi172.stdout:1 2024-05-08T16:56:55.918 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/1/new.json 2024-05-08T16:56:55.919 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=4457b0f1-028b-482b-8ee3-7ee05861e6cc --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81110/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDWrjtmqWLcORAAd8YQ32KdfmLYgoPNFGE3rw== --osd-uuid 0f51dcbd-8561-4314-b628-17fe241f145f 2024-05-08T16:56:55.955 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:56:55.956+0000 7f38148807c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:56:55.958 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:56:55.960+0000 7f38148807c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:56:55.959 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:56:55.960+0000 7f38148807c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:56:55.960 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:56:55.960+0000 7f38148807c0 -1 bluestore(td/test-erasure-eio/1/block) _read_bdev_label failed to open td/test-erasure-eio/1/block: (2) No such file or directory 2024-05-08T16:56:55.960 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:56:55.960+0000 7f38148807c0 -1 bluestore(td/test-erasure-eio/1/block) _read_bdev_label failed to open td/test-erasure-eio/1/block: (2) No such file or directory 2024-05-08T16:56:55.960 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:56:55.960+0000 7f38148807c0 -1 bluestore(td/test-erasure-eio/1/block) _read_bdev_label failed to open td/test-erasure-eio/1/block: (2) No such file or directory 2024-05-08T16:56:56.037 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:56:56.040+0000 7f38148807c0 -1 bluestore(td/test-erasure-eio/1) _read_fsid unparsable uuid 2024-05-08T16:57:02.942 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/1/keyring 2024-05-08T16:57:02.942 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-05-08T16:57:02.943 INFO:tasks.workunit.client.0.smithi172.stdout:adding osd1 key to auth repository 2024-05-08T16:57:02.943 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-05-08T16:57:02.943 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-05-08T16:57:03.497 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-05-08T16:57:03.497 INFO:tasks.workunit.client.0.smithi172.stdout:start osd.1 2024-05-08T16:57:03.498 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=4457b0f1-028b-482b-8ee3-7ee05861e6cc --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81110/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-05-08T16:57:03.498 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-05-08T16:57:03.498 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-05-08T16:57:03.498 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-05-08T16:57:03.530 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:57:03.532+0000 7fd238b087c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:57:03.532 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:57:03.532+0000 7fd238b087c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:57:03.534 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:57:03.536+0000 7fd238b087c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:57:03.949 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-05-08T16:57:03.949 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-08T16:57:03.949 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-05-08T16:57:03.949 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-08T16:57:03.949 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-08T16:57:03.950 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:57:03.950 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-08T16:57:03.950 INFO:tasks.workunit.client.0.smithi172.stdout:0 2024-05-08T16:57:03.950 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:57:03.950 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T16:57:04.110 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:57:04.112+0000 7fd238b087c0 -1 Falling back to public interface 2024-05-08T16:57:04.355 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:57:05.356 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:57:05.357 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:57:05.357 INFO:tasks.workunit.client.0.smithi172.stdout:1 2024-05-08T16:57:05.357 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-08T16:57:05.358 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:57:05.358 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T16:57:05.758 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:57:06.745 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:57:06.748+0000 7fd238b087c0 -1 osd.1 0 log_to_monitors true 2024-05-08T16:57:06.759 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:57:06.760 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:57:06.760 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-08T16:57:06.760 INFO:tasks.workunit.client.0.smithi172.stdout:2 2024-05-08T16:57:06.760 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:57:06.760 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T16:57:07.266 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:57:08.267 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:57:08.268 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:57:08.268 INFO:tasks.workunit.client.0.smithi172.stdout:3 2024-05-08T16:57:08.268 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-05-08T16:57:08.269 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:57:08.269 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T16:57:08.827 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:57:09.828 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:57:09.829 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:57:09.829 INFO:tasks.workunit.client.0.smithi172.stdout:4 2024-05-08T16:57:09.829 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-05-08T16:57:09.829 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:57:09.830 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T16:57:10.269 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:57:11.271 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:57:11.271 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:57:11.271 INFO:tasks.workunit.client.0.smithi172.stdout:5 2024-05-08T16:57:11.272 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-05-08T16:57:11.272 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:57:11.272 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T16:57:11.925 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:57:12.926 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:57:12.926 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:57:12.927 INFO:tasks.workunit.client.0.smithi172.stdout:6 2024-05-08T16:57:12.927 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-05-08T16:57:12.927 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:57:12.927 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T16:57:13.556 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:57:14.558 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:57:14.558 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:57:14.558 INFO:tasks.workunit.client.0.smithi172.stdout:7 2024-05-08T16:57:14.559 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-05-08T16:57:14.559 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:57:14.559 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T16:57:15.077 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:57:16.078 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:57:16.079 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:57:16.079 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-05-08T16:57:16.079 INFO:tasks.workunit.client.0.smithi172.stdout:8 2024-05-08T16:57:16.079 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:57:16.079 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T16:57:16.522 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:57:17.523 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:57:17.523 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:57:17.524 INFO:tasks.workunit.client.0.smithi172.stdout:9 2024-05-08T16:57:17.524 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-05-08T16:57:17.524 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:57:17.525 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T16:57:18.092 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:57:19.093 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:57:19.093 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:57:19.094 INFO:tasks.workunit.client.0.smithi172.stdout:10 2024-05-08T16:57:19.094 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-05-08T16:57:19.094 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:57:19.095 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T16:57:19.633 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:57:20.635 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:57:20.635 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:57:20.635 INFO:tasks.workunit.client.0.smithi172.stdout:11 2024-05-08T16:57:20.636 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-05-08T16:57:20.636 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:57:20.636 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T16:57:21.095 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:57:22.097 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:57:22.097 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:57:22.097 INFO:tasks.workunit.client.0.smithi172.stdout:12 2024-05-08T16:57:22.098 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-05-08T16:57:22.098 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:57:22.098 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T16:57:22.764 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:57:23.766 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:57:23.766 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:57:23.766 INFO:tasks.workunit.client.0.smithi172.stdout:13 2024-05-08T16:57:23.767 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-05-08T16:57:23.767 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:57:23.767 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T16:57:24.413 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:57:25.415 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:57:25.415 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:57:25.416 INFO:tasks.workunit.client.0.smithi172.stdout:14 2024-05-08T16:57:25.416 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-05-08T16:57:25.416 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:57:25.417 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T16:57:26.283 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:57:27.284 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:57:27.285 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:57:27.285 INFO:tasks.workunit.client.0.smithi172.stdout:15 2024-05-08T16:57:27.286 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-05-08T16:57:27.286 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:57:27.286 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T16:57:28.207 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:57:29.209 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:57:29.209 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:57:29.210 INFO:tasks.workunit.client.0.smithi172.stdout:16 2024-05-08T16:57:29.210 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-05-08T16:57:29.210 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:57:29.210 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T16:57:30.134 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:57:31.136 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:57:31.136 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:57:31.136 INFO:tasks.workunit.client.0.smithi172.stdout:17 2024-05-08T16:57:31.137 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-05-08T16:57:31.137 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:57:31.137 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T16:57:32.144 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:57:33.146 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:57:33.146 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:57:33.146 INFO:tasks.workunit.client.0.smithi172.stdout:18 2024-05-08T16:57:33.147 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-05-08T16:57:33.147 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:57:33.147 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T16:57:33.644 INFO:tasks.workunit.client.0.smithi172.stdout:osd.1 up in weight 1 up_from 13 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/1629254961,v1:127.0.0.1:6811/1629254961] [v2:127.0.0.1:6812/1629254961,v1:127.0.0.1:6813/1629254961] exists,up 0f51dcbd-8561-4314-b628-17fe241f145f 2024-05-08T16:57:33.648 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-08T16:57:33.648 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-08T16:57:33.648 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-08T16:57:33.648 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-05-08T16:57:33.648 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 2 2024-05-08T16:57:33.648 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-05-08T16:57:33.648 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-05-08T16:57:33.648 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-05-08T16:57:33.648 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-05-08T16:57:33.649 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/2 2024-05-08T16:57:33.649 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=4457b0f1-028b-482b-8ee3-7ee05861e6cc --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-05-08T16:57:33.649 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-08T16:57:33.649 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-05-08T16:57:33.649 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-08T16:57:33.649 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/2' 2024-05-08T16:57:33.649 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/2/journal' 2024-05-08T16:57:33.649 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-05-08T16:57:33.649 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-05-08T16:57:33.649 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-05-08T16:57:33.649 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-05-08T16:57:33.649 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-08T16:57:33.649 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-08T16:57:33.650 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-08T16:57:33.650 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T16:57:33.650 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81110 2024-05-08T16:57:33.650 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.81110/$cluster-$name.asok' 2024-05-08T16:57:33.650 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.81110/$cluster-$name.asok' 2024-05-08T16:57:33.650 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-05-08T16:57:33.650 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-05-08T16:57:33.650 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-05-08T16:57:33.650 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-05-08T16:57:33.650 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-05-08T16:57:33.650 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-08T16:57:33.650 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-08T16:57:33.650 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-08T16:57:33.650 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-08T16:57:33.651 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-05-08T16:57:33.651 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-05-08T16:57:33.651 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/2 2024-05-08T16:57:33.651 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-05-08T16:57:33.651 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=f32c06af-4770-4220-a6e6-c79c664ba00b 2024-05-08T16:57:33.652 INFO:tasks.workunit.client.0.smithi172.stdout:add osd2 f32c06af-4770-4220-a6e6-c79c664ba00b 2024-05-08T16:57:33.652 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 f32c06af-4770-4220-a6e6-c79c664ba00b' 2024-05-08T16:57:33.652 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-05-08T16:57:33.673 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQD9rjtmqRdoKBAA4Vi5crvu7roeHq7/KcdQgg== 2024-05-08T16:57:33.673 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQD9rjtmqRdoKBAA4Vi5crvu7roeHq7/KcdQgg=="}' 2024-05-08T16:57:33.673 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new f32c06af-4770-4220-a6e6-c79c664ba00b -i td/test-erasure-eio/2/new.json 2024-05-08T16:57:34.763 INFO:tasks.workunit.client.0.smithi172.stdout:2 2024-05-08T16:57:34.782 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/2/new.json 2024-05-08T16:57:34.783 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=4457b0f1-028b-482b-8ee3-7ee05861e6cc --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81110/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQD9rjtmqRdoKBAA4Vi5crvu7roeHq7/KcdQgg== --osd-uuid f32c06af-4770-4220-a6e6-c79c664ba00b 2024-05-08T16:57:34.820 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:57:34.823+0000 7ffaa92827c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:57:34.860 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:57:34.863+0000 7ffaa92827c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:57:34.861 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:57:34.867+0000 7ffaa92827c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:57:34.862 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:57:34.867+0000 7ffaa92827c0 -1 bluestore(td/test-erasure-eio/2/block) _read_bdev_label failed to open td/test-erasure-eio/2/block: (2) No such file or directory 2024-05-08T16:57:34.862 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:57:34.867+0000 7ffaa92827c0 -1 bluestore(td/test-erasure-eio/2/block) _read_bdev_label failed to open td/test-erasure-eio/2/block: (2) No such file or directory 2024-05-08T16:57:34.862 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:57:34.867+0000 7ffaa92827c0 -1 bluestore(td/test-erasure-eio/2/block) _read_bdev_label failed to open td/test-erasure-eio/2/block: (2) No such file or directory 2024-05-08T16:57:34.985 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:57:34.987+0000 7ffaa92827c0 -1 bluestore(td/test-erasure-eio/2) _read_fsid unparsable uuid 2024-05-08T16:57:41.914 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/2/keyring 2024-05-08T16:57:41.914 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-05-08T16:57:41.915 INFO:tasks.workunit.client.0.smithi172.stdout:adding osd2 key to auth repository 2024-05-08T16:57:41.916 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-05-08T16:57:41.916 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-05-08T16:57:42.477 INFO:tasks.workunit.client.0.smithi172.stdout:start osd.2 2024-05-08T16:57:42.477 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-05-08T16:57:42.478 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-05-08T16:57:42.478 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-05-08T16:57:42.478 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=4457b0f1-028b-482b-8ee3-7ee05861e6cc --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81110/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-05-08T16:57:42.478 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-05-08T16:57:42.503 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:57:42.507+0000 7f2b340907c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:57:42.505 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:57:42.511+0000 7f2b340907c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:57:42.506 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:57:42.511+0000 7f2b340907c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:57:42.884 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-05-08T16:57:42.884 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-08T16:57:42.884 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-05-08T16:57:42.884 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-08T16:57:42.884 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-08T16:57:42.884 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:57:42.885 INFO:tasks.workunit.client.0.smithi172.stdout:0 2024-05-08T16:57:42.885 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-08T16:57:42.885 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:57:42.885 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T16:57:43.046 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:57:43.051+0000 7f2b340907c0 -1 Falling back to public interface 2024-05-08T16:57:43.325 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:57:44.327 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:57:44.327 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:57:44.328 INFO:tasks.workunit.client.0.smithi172.stdout:1 2024-05-08T16:57:44.328 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-08T16:57:44.328 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:57:44.328 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T16:57:44.732 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:57:45.734 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:57:45.734 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:57:45.735 INFO:tasks.workunit.client.0.smithi172.stdout:2 2024-05-08T16:57:45.735 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-08T16:57:45.735 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:57:45.735 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T16:57:45.832 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:57:45.835+0000 7f2b340907c0 -1 osd.2 0 log_to_monitors true 2024-05-08T16:57:46.200 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:57:47.201 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:57:47.202 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:57:47.202 INFO:tasks.workunit.client.0.smithi172.stdout:3 2024-05-08T16:57:47.202 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-05-08T16:57:47.202 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:57:47.202 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T16:57:47.648 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:57:48.650 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:57:48.650 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:57:48.650 INFO:tasks.workunit.client.0.smithi172.stdout:4 2024-05-08T16:57:48.651 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-05-08T16:57:48.651 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:57:48.651 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T16:57:49.085 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:57:50.087 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:57:50.087 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:57:50.087 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-05-08T16:57:50.088 INFO:tasks.workunit.client.0.smithi172.stdout:5 2024-05-08T16:57:50.088 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:57:50.088 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T16:57:50.559 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:57:51.561 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:57:51.561 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:57:51.561 INFO:tasks.workunit.client.0.smithi172.stdout:6 2024-05-08T16:57:51.562 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-05-08T16:57:51.562 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:57:51.562 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T16:57:52.124 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:57:53.126 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:57:53.126 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:57:53.126 INFO:tasks.workunit.client.0.smithi172.stdout:7 2024-05-08T16:57:53.127 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-05-08T16:57:53.127 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:57:53.127 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T16:57:53.607 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:57:54.609 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:57:54.609 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:57:54.610 INFO:tasks.workunit.client.0.smithi172.stdout:8 2024-05-08T16:57:54.610 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-05-08T16:57:54.610 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:57:54.610 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T16:57:55.090 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:57:56.092 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:57:56.092 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:57:56.093 INFO:tasks.workunit.client.0.smithi172.stdout:9 2024-05-08T16:57:56.093 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-05-08T16:57:56.093 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:57:56.093 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T16:57:56.535 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:57:57.537 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:57:57.537 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:57:57.538 INFO:tasks.workunit.client.0.smithi172.stdout:10 2024-05-08T16:57:57.538 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-05-08T16:57:57.538 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:57:57.538 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T16:57:58.029 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:57:59.031 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:57:59.031 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:57:59.031 INFO:tasks.workunit.client.0.smithi172.stdout:11 2024-05-08T16:57:59.031 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-05-08T16:57:59.032 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:57:59.032 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T16:57:59.536 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:58:00.538 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:58:00.538 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:58:00.539 INFO:tasks.workunit.client.0.smithi172.stdout:12 2024-05-08T16:58:00.540 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-05-08T16:58:00.540 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:58:00.540 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T16:58:01.284 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:58:02.285 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:58:02.285 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:58:02.285 INFO:tasks.workunit.client.0.smithi172.stdout:13 2024-05-08T16:58:02.286 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-05-08T16:58:02.286 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:58:02.286 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T16:58:02.921 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:58:03.922 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:58:03.923 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:58:03.923 INFO:tasks.workunit.client.0.smithi172.stdout:14 2024-05-08T16:58:03.923 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-05-08T16:58:03.924 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:58:03.924 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T16:58:04.775 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:58:05.777 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:58:05.777 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:58:05.777 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-05-08T16:58:05.777 INFO:tasks.workunit.client.0.smithi172.stdout:15 2024-05-08T16:58:05.778 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:58:05.778 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T16:58:06.681 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:58:07.684 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:58:07.684 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:58:07.684 INFO:tasks.workunit.client.0.smithi172.stdout:16 2024-05-08T16:58:07.685 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-05-08T16:58:07.685 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:58:07.685 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T16:58:08.594 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:58:09.596 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:58:09.597 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:58:09.597 INFO:tasks.workunit.client.0.smithi172.stdout:17 2024-05-08T16:58:09.597 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-05-08T16:58:09.597 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:58:09.597 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T16:58:10.136 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:58:11.138 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:58:11.138 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:58:11.138 INFO:tasks.workunit.client.0.smithi172.stdout:18 2024-05-08T16:58:11.139 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-05-08T16:58:11.139 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:58:11.139 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T16:58:11.923 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:58:12.924 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:58:12.924 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:58:12.924 INFO:tasks.workunit.client.0.smithi172.stdout:19 2024-05-08T16:58:12.925 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-05-08T16:58:12.925 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:58:12.925 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T16:58:13.333 INFO:tasks.workunit.client.0.smithi172.stdout:osd.2 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/3815033671,v1:127.0.0.1:6819/3815033671] [v2:127.0.0.1:6820/3815033671,v1:127.0.0.1:6821/3815033671] exists,up f32c06af-4770-4220-a6e6-c79c664ba00b 2024-05-08T16:58:13.335 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-08T16:58:13.336 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-08T16:58:13.336 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-08T16:58:13.336 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-05-08T16:58:13.336 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 3 2024-05-08T16:58:13.336 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-05-08T16:58:13.336 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-05-08T16:58:13.336 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2024-05-08T16:58:13.336 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-05-08T16:58:13.336 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/3 2024-05-08T16:58:13.336 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=4457b0f1-028b-482b-8ee3-7ee05861e6cc --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-05-08T16:58:13.336 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-08T16:58:13.336 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-05-08T16:58:13.336 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-08T16:58:13.337 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/3' 2024-05-08T16:58:13.337 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/3/journal' 2024-05-08T16:58:13.337 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-05-08T16:58:13.337 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-05-08T16:58:13.337 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-05-08T16:58:13.337 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-05-08T16:58:13.337 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-08T16:58:13.337 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-08T16:58:13.337 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-08T16:58:13.337 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T16:58:13.337 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81110 2024-05-08T16:58:13.337 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.81110/$cluster-$name.asok' 2024-05-08T16:58:13.339 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.81110/$cluster-$name.asok' 2024-05-08T16:58:13.339 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-05-08T16:58:13.339 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-05-08T16:58:13.339 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-05-08T16:58:13.339 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-05-08T16:58:13.340 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-05-08T16:58:13.340 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-08T16:58:13.340 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-08T16:58:13.340 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-08T16:58:13.340 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-08T16:58:13.340 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-05-08T16:58:13.340 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-05-08T16:58:13.340 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/3 2024-05-08T16:58:13.340 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-05-08T16:58:13.340 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=724678d7-ca35-4cdd-a1aa-910a2045b6f4 2024-05-08T16:58:13.340 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 724678d7-ca35-4cdd-a1aa-910a2045b6f4' 2024-05-08T16:58:13.341 INFO:tasks.workunit.client.0.smithi172.stdout:add osd3 724678d7-ca35-4cdd-a1aa-910a2045b6f4 2024-05-08T16:58:13.341 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-05-08T16:58:13.360 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAlrztmX0PUFRAAq60YmDy7hHQ6fxbDm6jw3w== 2024-05-08T16:58:13.361 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAlrztmX0PUFRAAq60YmDy7hHQ6fxbDm6jw3w=="}' 2024-05-08T16:58:13.361 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 724678d7-ca35-4cdd-a1aa-910a2045b6f4 -i td/test-erasure-eio/3/new.json 2024-05-08T16:58:13.823 INFO:tasks.workunit.client.0.smithi172.stdout:3 2024-05-08T16:58:13.840 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/3/new.json 2024-05-08T16:58:13.841 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=4457b0f1-028b-482b-8ee3-7ee05861e6cc --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81110/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAlrztmX0PUFRAAq60YmDy7hHQ6fxbDm6jw3w== --osd-uuid 724678d7-ca35-4cdd-a1aa-910a2045b6f4 2024-05-08T16:58:13.875 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:58:13.878+0000 7eff61dab7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:58:14.127 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:58:14.130+0000 7eff61dab7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:58:14.129 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:58:14.134+0000 7eff61dab7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:58:14.130 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:58:14.134+0000 7eff61dab7c0 -1 bluestore(td/test-erasure-eio/3/block) _read_bdev_label failed to open td/test-erasure-eio/3/block: (2) No such file or directory 2024-05-08T16:58:14.130 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:58:14.134+0000 7eff61dab7c0 -1 bluestore(td/test-erasure-eio/3/block) _read_bdev_label failed to open td/test-erasure-eio/3/block: (2) No such file or directory 2024-05-08T16:58:14.130 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:58:14.134+0000 7eff61dab7c0 -1 bluestore(td/test-erasure-eio/3/block) _read_bdev_label failed to open td/test-erasure-eio/3/block: (2) No such file or directory 2024-05-08T16:58:14.293 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:58:14.298+0000 7eff61dab7c0 -1 bluestore(td/test-erasure-eio/3) _read_fsid unparsable uuid 2024-05-08T16:58:22.136 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/3/keyring 2024-05-08T16:58:22.136 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-05-08T16:58:22.137 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2024-05-08T16:58:22.137 INFO:tasks.workunit.client.0.smithi172.stdout:adding osd3 key to auth repository 2024-05-08T16:58:22.137 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-05-08T16:58:22.698 INFO:tasks.workunit.client.0.smithi172.stdout:start osd.3 2024-05-08T16:58:22.699 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2024-05-08T16:58:22.699 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=4457b0f1-028b-482b-8ee3-7ee05861e6cc --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81110/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-05-08T16:58:22.700 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-05-08T16:58:22.700 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-05-08T16:58:22.700 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-05-08T16:58:22.730 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:58:22.734+0000 7f8ac042d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:58:22.732 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:58:22.734+0000 7f8ac042d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:58:22.733 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:58:22.738+0000 7f8ac042d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:58:23.115 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2024-05-08T16:58:23.116 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-08T16:58:23.116 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-05-08T16:58:23.116 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-08T16:58:23.116 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-08T16:58:23.116 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:58:23.116 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-08T16:58:23.116 INFO:tasks.workunit.client.0.smithi172.stdout:0 2024-05-08T16:58:23.117 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:58:23.117 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T16:58:23.298 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:58:23.302+0000 7f8ac042d7c0 -1 Falling back to public interface 2024-05-08T16:58:23.536 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:58:24.537 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:58:24.538 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:58:24.538 INFO:tasks.workunit.client.0.smithi172.stdout:1 2024-05-08T16:58:24.538 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-08T16:58:24.538 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:58:24.539 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T16:58:24.968 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:58:25.480 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:58:25.482+0000 7f8ac042d7c0 -1 osd.3 0 log_to_monitors true 2024-05-08T16:58:25.970 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:58:25.970 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:58:25.970 INFO:tasks.workunit.client.0.smithi172.stdout:2 2024-05-08T16:58:25.971 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-08T16:58:25.971 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:58:25.971 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T16:58:26.571 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:58:27.572 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:58:27.572 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:58:27.573 INFO:tasks.workunit.client.0.smithi172.stdout:3 2024-05-08T16:58:27.573 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-05-08T16:58:27.573 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:58:27.573 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T16:58:28.377 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:58:29.378 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:58:29.379 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:58:29.379 INFO:tasks.workunit.client.0.smithi172.stdout:4 2024-05-08T16:58:29.379 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-05-08T16:58:29.380 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:58:29.380 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T16:58:30.020 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:58:31.021 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:58:31.022 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:58:31.022 INFO:tasks.workunit.client.0.smithi172.stdout:5 2024-05-08T16:58:31.023 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-05-08T16:58:31.023 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:58:31.023 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T16:58:31.571 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:58:32.573 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:58:32.574 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:58:32.574 INFO:tasks.workunit.client.0.smithi172.stdout:6 2024-05-08T16:58:32.574 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-05-08T16:58:32.574 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:58:32.575 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T16:58:33.032 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:58:34.034 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:58:34.034 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:58:34.035 INFO:tasks.workunit.client.0.smithi172.stdout:7 2024-05-08T16:58:34.035 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-05-08T16:58:34.035 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:58:34.035 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T16:58:34.550 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:58:35.551 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:58:35.552 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:58:35.552 INFO:tasks.workunit.client.0.smithi172.stdout:8 2024-05-08T16:58:35.552 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-05-08T16:58:35.552 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:58:35.553 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T16:58:36.028 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:58:37.030 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:58:37.030 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:58:37.031 INFO:tasks.workunit.client.0.smithi172.stdout:9 2024-05-08T16:58:37.031 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-05-08T16:58:37.031 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:58:37.031 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T16:58:37.508 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:58:38.509 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:58:38.510 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:58:38.510 INFO:tasks.workunit.client.0.smithi172.stdout:10 2024-05-08T16:58:38.510 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-05-08T16:58:38.510 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:58:38.510 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T16:58:38.953 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:58:39.954 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:58:39.954 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:58:39.955 INFO:tasks.workunit.client.0.smithi172.stdout:11 2024-05-08T16:58:39.955 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-05-08T16:58:39.955 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:58:39.955 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T16:58:40.467 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:58:41.469 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:58:41.469 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:58:41.469 INFO:tasks.workunit.client.0.smithi172.stdout:12 2024-05-08T16:58:41.470 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-05-08T16:58:41.470 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:58:41.470 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T16:58:42.644 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:58:43.646 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:58:43.646 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:58:43.647 INFO:tasks.workunit.client.0.smithi172.stdout:13 2024-05-08T16:58:43.647 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-05-08T16:58:43.647 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:58:43.647 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T16:58:44.605 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:58:45.606 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:58:45.607 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:58:45.607 INFO:tasks.workunit.client.0.smithi172.stdout:14 2024-05-08T16:58:45.607 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-05-08T16:58:45.607 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:58:45.607 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T16:58:46.569 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:58:47.571 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:58:47.571 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:58:47.571 INFO:tasks.workunit.client.0.smithi172.stdout:15 2024-05-08T16:58:47.572 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-05-08T16:58:47.572 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:58:47.572 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T16:58:48.652 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:58:49.654 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:58:49.654 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:58:49.654 INFO:tasks.workunit.client.0.smithi172.stdout:16 2024-05-08T16:58:49.655 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-05-08T16:58:49.655 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:58:49.655 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T16:58:50.206 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:58:51.208 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:58:51.208 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:58:51.209 INFO:tasks.workunit.client.0.smithi172.stdout:17 2024-05-08T16:58:51.209 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-05-08T16:58:51.209 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:58:51.209 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T16:58:52.455 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:58:53.457 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:58:53.457 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:58:53.458 INFO:tasks.workunit.client.0.smithi172.stdout:18 2024-05-08T16:58:53.458 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-05-08T16:58:53.458 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:58:53.458 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T16:58:54.235 INFO:tasks.workunit.client.0.smithi172.stdout:osd.3 up in weight 1 up_from 28 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/1299974432,v1:127.0.0.1:6827/1299974432] [v2:127.0.0.1:6828/1299974432,v1:127.0.0.1:6829/1299974432] exists,up 724678d7-ca35-4cdd-a1aa-910a2045b6f4 2024-05-08T16:58:54.235 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-08T16:58:54.235 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-08T16:58:54.235 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-08T16:58:54.236 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: get_asok_path osd.0 2024-05-08T16:58:54.236 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-05-08T16:58:54.236 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-05-08T16:58:54.237 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-08T16:58:54.237 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T16:58:54.237 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81110 2024-05-08T16:58:54.237 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.81110/ceph-osd.0.asok 2024-05-08T16:58:54.237 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: CEPH_ARGS= 2024-05-08T16:58:54.237 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: ceph --admin-daemon /tmp/ceph-asok.81110/ceph-osd.0.asok log flush 2024-05-08T16:58:54.352 INFO:tasks.workunit.client.0.smithi172.stdout:{} 2024-05-08T16:58:54.365 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:56: setup_osds: grep 'load: jerasure.*lrc' td/test-erasure-eio/osd.0.log 2024-05-08T16:58:54.370 INFO:tasks.workunit.client.0.smithi172.stdout:2024-05-08T16:56:26.184+0000 7f4f25b677c0 0 load: jerasure load: lrc 2024-05-08T16:58:54.373 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:305: TEST_rados_get_bad_size_shard_0: local poolname=pool-jerasure 2024-05-08T16:58:54.373 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:306: TEST_rados_get_bad_size_shard_0: create_erasure_coded_pool pool-jerasure 2 1 2024-05-08T16:58:54.373 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:67: create_erasure_coded_pool: local poolname=pool-jerasure 2024-05-08T16:58:54.373 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:68: create_erasure_coded_pool: shift 2024-05-08T16:58:54.373 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:69: create_erasure_coded_pool: local k=2 2024-05-08T16:58:54.373 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:70: create_erasure_coded_pool: shift 2024-05-08T16:58:54.373 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:71: create_erasure_coded_pool: local m=1 2024-05-08T16:58:54.373 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:72: create_erasure_coded_pool: shift 2024-05-08T16:58:54.374 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:74: create_erasure_coded_pool: ceph osd erasure-code-profile set myprofile plugin=jerasure k=2 m=1 crush-failure-domain=osd 2024-05-08T16:58:55.322 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:78: create_erasure_coded_pool: create_pool pool-jerasure 1 1 erasure myprofile 2024-05-08T16:58:55.322 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create pool-jerasure 1 1 erasure myprofile 2024-05-08T16:58:56.537 INFO:tasks.workunit.client.0.smithi172.stderr:pool 'pool-jerasure' already exists 2024-05-08T16:58:56.554 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-05-08T16:58:57.557 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:80: create_erasure_coded_pool: wait_for_clean 2024-05-08T16:58:57.557 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-05-08T16:58:57.557 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-05-08T16:58:57.557 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-05-08T16:58:57.557 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-05-08T16:58:57.558 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-05-08T16:58:57.558 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-05-08T16:58:57.559 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-05-08T16:58:57.559 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-05-08T16:58:57.559 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-05-08T16:58:57.685 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-05-08T16:58:57.685 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-05-08T16:58:57.685 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-05-08T16:58:57.685 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-05-08T16:58:57.685 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-05-08T16:58:57.685 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-05-08T16:58:58.173 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-05-08T16:58:58.173 INFO:tasks.workunit.client.0.smithi172.stderr:1 2024-05-08T16:58:58.173 INFO:tasks.workunit.client.0.smithi172.stderr:2 2024-05-08T16:58:58.174 INFO:tasks.workunit.client.0.smithi172.stderr:3' 2024-05-08T16:58:58.174 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-05-08T16:58:58.174 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T16:58:58.174 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-05-08T16:58:58.347 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=25769803808 2024-05-08T16:58:58.348 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 25769803808 2024-05-08T16:58:58.348 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803808' 2024-05-08T16:58:58.348 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T16:58:58.348 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-05-08T16:58:58.497 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=55834574875 2024-05-08T16:58:58.498 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 55834574875 2024-05-08T16:58:58.498 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803808 1-55834574875' 2024-05-08T16:58:58.498 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T16:58:58.498 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-05-08T16:58:58.648 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=85899345937 2024-05-08T16:58:58.648 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 85899345937 2024-05-08T16:58:58.648 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803808 1-55834574875 2-85899345937' 2024-05-08T16:58:58.648 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T16:58:58.649 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-05-08T16:58:58.787 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=120259084298 2024-05-08T16:58:58.787 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 120259084298 2024-05-08T16:58:58.787 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803808 1-55834574875 2-85899345937 3-120259084298' 2024-05-08T16:58:58.787 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-08T16:58:58.787 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-25769803808 2024-05-08T16:58:58.787 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-08T16:58:58.788 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-05-08T16:58:58.789 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-25769803808 2024-05-08T16:58:58.789 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-08T16:58:58.790 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=25769803808 2024-05-08T16:58:58.790 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 25769803808' 2024-05-08T16:58:58.790 INFO:tasks.workunit.client.0.smithi172.stdout:waiting osd.0 seq 25769803808 2024-05-08T16:58:58.790 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-05-08T16:58:59.173 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 25769803808 -lt 25769803808 2024-05-08T16:58:59.173 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-08T16:58:59.173 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-55834574875 2024-05-08T16:58:59.174 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-08T16:58:59.175 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-05-08T16:58:59.176 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-55834574875 2024-05-08T16:58:59.176 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-08T16:58:59.177 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=55834574875 2024-05-08T16:58:59.177 INFO:tasks.workunit.client.0.smithi172.stdout:waiting osd.1 seq 55834574875 2024-05-08T16:58:59.177 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 55834574875' 2024-05-08T16:58:59.178 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-05-08T16:58:59.641 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 55834574875 -lt 55834574875 2024-05-08T16:58:59.641 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-08T16:58:59.642 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-85899345937 2024-05-08T16:58:59.642 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-08T16:58:59.644 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-05-08T16:58:59.645 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-85899345937 2024-05-08T16:58:59.645 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-08T16:58:59.646 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=85899345937 2024-05-08T16:58:59.646 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 85899345937' 2024-05-08T16:58:59.646 INFO:tasks.workunit.client.0.smithi172.stdout:waiting osd.2 seq 85899345937 2024-05-08T16:58:59.646 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-05-08T16:59:00.138 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 85899345937 -lt 85899345937 2024-05-08T16:59:00.138 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-08T16:59:00.139 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-120259084298 2024-05-08T16:59:00.139 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-08T16:59:00.141 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-05-08T16:59:00.142 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-120259084298 2024-05-08T16:59:00.142 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-08T16:59:00.143 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=120259084298 2024-05-08T16:59:00.143 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 120259084298' 2024-05-08T16:59:00.144 INFO:tasks.workunit.client.0.smithi172.stdout:waiting osd.3 seq 120259084298 2024-05-08T16:59:00.144 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-05-08T16:59:00.544 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 120259084296 -lt 120259084298 2024-05-08T16:59:00.544 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-05-08T16:59:01.546 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-05-08T16:59:01.546 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-05-08T16:59:01.954 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 120259084299 -lt 120259084298 2024-05-08T16:59:01.955 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-05-08T16:59:01.956 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-05-08T16:59:01.956 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-05-08T16:59:02.499 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-05-08T16:59:02.500 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-05-08T16:59:02.500 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-05-08T16:59:02.500 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-05-08T16:59:02.500 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-05-08T16:59:02.500 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-05-08T16:59:02.500 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-05-08T16:59:02.500 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-05-08T16:59:02.886 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-05-08T16:59:02.887 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-05-08T16:59:02.887 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-05-08T16:59:02.888 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-05-08T16:59:03.394 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-05-08T16:59:03.395 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-05-08T16:59:03.395 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-05-08T16:59:03.395 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:308: TEST_rados_get_bad_size_shard_0: local shard_id=0 2024-05-08T16:59:03.395 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:309: TEST_rados_get_bad_size_shard_0: rados_get_data_bad_size td/test-erasure-eio 0 10 2024-05-08T16:59:03.395 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:222: rados_get_data_bad_size: local dir=td/test-erasure-eio 2024-05-08T16:59:03.395 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:223: rados_get_data_bad_size: shift 2024-05-08T16:59:03.395 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:224: rados_get_data_bad_size: local shard_id=0 2024-05-08T16:59:03.395 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:225: rados_get_data_bad_size: shift 2024-05-08T16:59:03.395 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:226: rados_get_data_bad_size: local bytes=10 2024-05-08T16:59:03.395 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:227: rados_get_data_bad_size: shift 2024-05-08T16:59:03.395 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:228: rados_get_data_bad_size: local mode=set 2024-05-08T16:59:03.395 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:230: rados_get_data_bad_size: local poolname=pool-jerasure 2024-05-08T16:59:03.395 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:231: rados_get_data_bad_size: local objname=obj-size-81110-0-10 2024-05-08T16:59:03.395 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:232: rados_get_data_bad_size: rados_put td/test-erasure-eio pool-jerasure obj-size-81110-0-10 2024-05-08T16:59:03.396 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:90: rados_put: local dir=td/test-erasure-eio 2024-05-08T16:59:03.396 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:91: rados_put: local poolname=pool-jerasure 2024-05-08T16:59:03.396 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:92: rados_put: local objname=obj-size-81110-0-10 2024-05-08T16:59:03.396 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-05-08T16:59:03.396 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 AAA 2024-05-08T16:59:03.396 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-05-08T16:59:03.396 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 BBB 2024-05-08T16:59:03.396 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-05-08T16:59:03.396 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 CCCC 2024-05-08T16:59:03.396 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-05-08T16:59:03.396 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 DDDD 2024-05-08T16:59:03.396 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:100: rados_put: rados --pool pool-jerasure put obj-size-81110-0-10 td/test-erasure-eio/ORIGINAL 2024-05-08T16:59:03.611 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:236: rados_get_data_bad_size: set_size obj-size-81110-0-10 td/test-erasure-eio 0 10 set 2024-05-08T16:59:03.611 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:192: set_size: local objname=obj-size-81110-0-10 2024-05-08T16:59:03.611 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:193: set_size: shift 2024-05-08T16:59:03.611 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:194: set_size: local dir=td/test-erasure-eio 2024-05-08T16:59:03.612 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:195: set_size: shift 2024-05-08T16:59:03.612 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:196: set_size: local shard_id=0 2024-05-08T16:59:03.612 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:197: set_size: shift 2024-05-08T16:59:03.612 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:198: set_size: local bytes=10 2024-05-08T16:59:03.612 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:199: set_size: shift 2024-05-08T16:59:03.612 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:200: set_size: local mode=set 2024-05-08T16:59:03.612 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:202: set_size: local poolname=pool-jerasure 2024-05-08T16:59:03.612 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: get_osds pool-jerasure obj-size-81110-0-10 2024-05-08T16:59:03.612 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2024-05-08T16:59:03.612 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-size-81110-0-10 2024-05-08T16:59:03.613 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure obj-size-81110-0-10 2024-05-08T16:59:03.613 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-05-08T16:59:04.011 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-05-08T16:59:04.012 INFO:tasks.workunit.client.0.smithi172.stderr:1 2024-05-08T16:59:04.012 INFO:tasks.workunit.client.0.smithi172.stderr:2' 2024-05-08T16:59:04.012 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 2 2024-05-08T16:59:04.012 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: initial_osds=('3' '1' '2') 2024-05-08T16:59:04.012 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: local -a initial_osds 2024-05-08T16:59:04.012 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:204: set_size: local osd_id=3 2024-05-08T16:59:04.012 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:205: set_size: ceph osd set noout 2024-05-08T16:59:04.701 INFO:tasks.workunit.client.0.smithi172.stderr:noout is set 2024-05-08T16:59:04.725 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:206: set_size: '[' set = add ']' 2024-05-08T16:59:04.725 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:210: set_size: '[' 10 = 0 ']' 2024-05-08T16:59:04.725 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:214: set_size: dd if=/dev/urandom bs=10 count=1 of=td/test-erasure-eio/CORRUPT 2024-05-08T16:59:04.726 INFO:tasks.workunit.client.0.smithi172.stderr:1+0 records in 2024-05-08T16:59:04.726 INFO:tasks.workunit.client.0.smithi172.stderr:1+0 records out 2024-05-08T16:59:04.726 INFO:tasks.workunit.client.0.smithi172.stderr:10 bytes copied, 0.000135626 s, 73.7 kB/s 2024-05-08T16:59:04.727 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:216: set_size: objectstore_tool td/test-erasure-eio 3 obj-size-81110-0-10 set-bytes td/test-erasure-eio/CORRUPT 2024-05-08T16:59:04.727 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/test-erasure-eio 2024-05-08T16:59:04.727 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-05-08T16:59:04.727 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=3 2024-05-08T16:59:04.727 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-05-08T16:59:04.727 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/test-erasure-eio 3 obj-size-81110-0-10 set-bytes td/test-erasure-eio/CORRUPT 2024-05-08T16:59:04.727 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/test-erasure-eio 2024-05-08T16:59:04.728 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-05-08T16:59:04.728 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=3 2024-05-08T16:59:04.728 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-05-08T16:59:04.728 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/test-erasure-eio TERM osd.3 2024-05-08T16:59:04.728 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-08T16:59:04.729 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-08T16:59:04.729 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-08T16:59:04.729 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-08T16:59:04.729 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-08T16:59:05.039 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-08T16:59:05.039 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/test-erasure-eio 3 obj-size-81110-0-10 set-bytes td/test-erasure-eio/CORRUPT 2024-05-08T16:59:05.039 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/test-erasure-eio 2024-05-08T16:59:05.039 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-05-08T16:59:05.039 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=3 2024-05-08T16:59:05.039 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-05-08T16:59:05.039 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/test-erasure-eio/3 2024-05-08T16:59:05.039 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/test-erasure-eio/3 obj-size-81110-0-10 set-bytes td/test-erasure-eio/CORRUPT 2024-05-08T16:59:07.607 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/test-erasure-eio 3 2024-05-08T16:59:07.607 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/test-erasure-eio 2024-05-08T16:59:07.608 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-05-08T16:59:07.608 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2024-05-08T16:59:07.608 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-05-08T16:59:07.608 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/test-erasure-eio/3 2024-05-08T16:59:07.608 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=4457b0f1-028b-482b-8ee3-7ee05861e6cc --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-05-08T16:59:07.608 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-08T16:59:07.608 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-05-08T16:59:07.608 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-08T16:59:07.608 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/3' 2024-05-08T16:59:07.608 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/3/journal' 2024-05-08T16:59:07.608 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-05-08T16:59:07.609 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-05-08T16:59:07.609 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-05-08T16:59:07.609 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-05-08T16:59:07.609 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-08T16:59:07.609 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-08T16:59:07.611 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-08T16:59:07.611 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T16:59:07.611 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81110 2024-05-08T16:59:07.611 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.81110/$cluster-$name.asok' 2024-05-08T16:59:07.611 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.81110/$cluster-$name.asok' 2024-05-08T16:59:07.611 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-05-08T16:59:07.611 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-05-08T16:59:07.612 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-05-08T16:59:07.612 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-08T16:59:07.612 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-08T16:59:07.612 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-08T16:59:07.612 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-08T16:59:07.612 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-05-08T16:59:07.612 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-05-08T16:59:07.612 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/test-erasure-eio/3 2024-05-08T16:59:07.612 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2024-05-08T16:59:07.612 INFO:tasks.workunit.client.0.smithi172.stderr:start osd.3 2024-05-08T16:59:07.612 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=4457b0f1-028b-482b-8ee3-7ee05861e6cc --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81110/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-05-08T16:59:07.613 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/test-erasure-eio/3/whoami 2024-05-08T16:59:07.613 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2024-05-08T16:59:07.614 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-05-08T16:59:07.614 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-05-08T16:59:07.614 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-05-08T16:59:07.649 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:59:07.653+0000 7feb712d47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:59:07.651 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:59:07.653+0000 7feb712d47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:59:07.653 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:59:07.657+0000 7feb712d47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:59:08.031 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2024-05-08T16:59:08.031 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-08T16:59:08.031 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-05-08T16:59:08.031 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-08T16:59:08.032 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-08T16:59:08.032 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:59:08.032 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-08T16:59:08.032 INFO:tasks.workunit.client.0.smithi172.stderr:0 2024-05-08T16:59:08.032 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T16:59:08.032 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:59:08.190 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:59:08.193+0000 7feb712d47c0 -1 Falling back to public interface 2024-05-08T16:59:08.490 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:59:09.491 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:59:09.492 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:59:09.492 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-08T16:59:09.492 INFO:tasks.workunit.client.0.smithi172.stderr:1 2024-05-08T16:59:09.492 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:59:09.492 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T16:59:09.912 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:59:10.914 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:59:10.914 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:59:10.914 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-08T16:59:10.914 INFO:tasks.workunit.client.0.smithi172.stderr:2 2024-05-08T16:59:10.915 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:59:10.915 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T16:59:11.067 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:59:11.069+0000 7feb712d47c0 -1 osd.3 36 log_to_monitors true 2024-05-08T16:59:11.475 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:59:12.478 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:59:12.478 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:59:12.478 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-05-08T16:59:12.478 INFO:tasks.workunit.client.0.smithi172.stderr:3 2024-05-08T16:59:12.478 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:59:12.478 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T16:59:13.210 INFO:tasks.workunit.client.0.smithi172.stderr:osd.3 up in weight 1 up_from 42 up_thru 42 down_at 39 last_clean_interval [28,36) [v2:127.0.0.1:6826/2235535918,v1:127.0.0.1:6827/2235535918] [v2:127.0.0.1:6828/2235535918,v1:127.0.0.1:6829/2235535918] exists,up 724678d7-ca35-4cdd-a1aa-910a2045b6f4 2024-05-08T16:59:13.210 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-08T16:59:13.210 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-08T16:59:13.211 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-08T16:59:13.211 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-05-08T16:59:13.211 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-05-08T16:59:13.211 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-05-08T16:59:13.211 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-05-08T16:59:13.211 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-05-08T16:59:13.211 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-05-08T16:59:13.211 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-05-08T16:59:13.211 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-05-08T16:59:13.211 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-05-08T16:59:13.211 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-05-08T16:59:13.327 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-05-08T16:59:13.327 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-05-08T16:59:13.327 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-05-08T16:59:13.327 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-05-08T16:59:13.327 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-05-08T16:59:13.327 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-05-08T16:59:13.848 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-05-08T16:59:13.848 INFO:tasks.workunit.client.0.smithi172.stderr:1 2024-05-08T16:59:13.848 INFO:tasks.workunit.client.0.smithi172.stderr:2 2024-05-08T16:59:13.848 INFO:tasks.workunit.client.0.smithi172.stderr:3' 2024-05-08T16:59:13.848 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-05-08T16:59:13.849 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T16:59:13.849 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-05-08T16:59:14.061 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=25769803813 2024-05-08T16:59:14.061 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 25769803813 2024-05-08T16:59:14.061 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803813' 2024-05-08T16:59:14.062 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T16:59:14.062 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-05-08T16:59:14.212 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=55834574880 2024-05-08T16:59:14.212 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 55834574880 2024-05-08T16:59:14.213 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803813 1-55834574880' 2024-05-08T16:59:14.213 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T16:59:14.213 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-05-08T16:59:14.364 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=85899345942 2024-05-08T16:59:14.364 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 85899345942 2024-05-08T16:59:14.365 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803813 1-55834574880 2-85899345942' 2024-05-08T16:59:14.365 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T16:59:14.365 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-05-08T16:59:14.517 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=180388626435 2024-05-08T16:59:14.517 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 180388626435 2024-05-08T16:59:14.517 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803813 1-55834574880 2-85899345942 3-180388626435' 2024-05-08T16:59:14.517 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-08T16:59:14.517 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-25769803813 2024-05-08T16:59:14.518 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-08T16:59:14.518 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-05-08T16:59:14.519 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-25769803813 2024-05-08T16:59:14.519 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-08T16:59:14.520 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=25769803813 2024-05-08T16:59:14.520 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 25769803813' 2024-05-08T16:59:14.520 INFO:tasks.workunit.client.0.smithi172.stderr:waiting osd.0 seq 25769803813 2024-05-08T16:59:14.521 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-05-08T16:59:14.941 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 25769803813 -lt 25769803813 2024-05-08T16:59:14.941 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-08T16:59:14.942 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-55834574880 2024-05-08T16:59:14.942 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-08T16:59:14.954 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-05-08T16:59:14.954 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-55834574880 2024-05-08T16:59:14.954 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-08T16:59:14.955 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=55834574880 2024-05-08T16:59:14.955 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 55834574880' 2024-05-08T16:59:14.955 INFO:tasks.workunit.client.0.smithi172.stderr:waiting osd.1 seq 55834574880 2024-05-08T16:59:14.955 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-05-08T16:59:15.370 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 55834574881 -lt 55834574880 2024-05-08T16:59:15.370 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-08T16:59:15.371 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-85899345942 2024-05-08T16:59:15.371 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-08T16:59:15.373 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-05-08T16:59:15.374 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-85899345942 2024-05-08T16:59:15.374 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-08T16:59:15.376 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=85899345942 2024-05-08T16:59:15.376 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 85899345942' 2024-05-08T16:59:15.376 INFO:tasks.workunit.client.0.smithi172.stderr:waiting osd.2 seq 85899345942 2024-05-08T16:59:15.376 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-05-08T16:59:15.781 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 85899345942 -lt 85899345942 2024-05-08T16:59:15.781 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-08T16:59:15.782 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-180388626435 2024-05-08T16:59:15.782 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-08T16:59:15.784 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-05-08T16:59:15.785 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-180388626435 2024-05-08T16:59:15.785 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-08T16:59:15.786 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=180388626435 2024-05-08T16:59:15.786 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 180388626435' 2024-05-08T16:59:15.786 INFO:tasks.workunit.client.0.smithi172.stderr:waiting osd.3 seq 180388626435 2024-05-08T16:59:15.787 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-05-08T16:59:16.221 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 180388626435 -lt 180388626435 2024-05-08T16:59:16.222 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-05-08T16:59:16.222 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-05-08T16:59:16.223 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-05-08T16:59:16.746 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-05-08T16:59:16.746 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-05-08T16:59:16.747 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-05-08T16:59:16.747 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-05-08T16:59:16.748 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-05-08T16:59:16.748 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-05-08T16:59:16.748 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-05-08T16:59:16.748 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-05-08T16:59:17.135 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-05-08T16:59:17.135 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-05-08T16:59:17.135 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-05-08T16:59:17.135 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-05-08T16:59:17.714 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-05-08T16:59:17.715 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-05-08T16:59:17.715 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-05-08T16:59:17.715 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:217: set_size: rm -f td/test-erasure-eio/CORRUPT 2024-05-08T16:59:17.715 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:218: set_size: ceph osd unset noout 2024-05-08T16:59:18.403 INFO:tasks.workunit.client.0.smithi172.stderr:noout is unset 2024-05-08T16:59:18.424 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:238: rados_get_data_bad_size: rados_get td/test-erasure-eio pool-jerasure obj-size-81110-0-10 2024-05-08T16:59:18.425 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2024-05-08T16:59:18.425 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2024-05-08T16:59:18.425 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=obj-size-81110-0-10 2024-05-08T16:59:18.425 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=ok 2024-05-08T16:59:18.425 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' ok = fail ']' 2024-05-08T16:59:18.425 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:120: rados_get: rados --pool pool-jerasure get obj-size-81110-0-10 td/test-erasure-eio/COPY 2024-05-08T16:59:18.678 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:121: rados_get: diff td/test-erasure-eio/ORIGINAL td/test-erasure-eio/COPY 2024-05-08T16:59:18.680 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:122: rados_get: rm td/test-erasure-eio/COPY 2024-05-08T16:59:18.682 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:241: rados_get_data_bad_size: expr 0 + 1 2024-05-08T16:59:18.684 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:241: rados_get_data_bad_size: shard_id=1 2024-05-08T16:59:18.684 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:242: rados_get_data_bad_size: set_size obj-size-81110-0-10 td/test-erasure-eio 1 10 set 2024-05-08T16:59:18.684 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:192: set_size: local objname=obj-size-81110-0-10 2024-05-08T16:59:18.684 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:193: set_size: shift 2024-05-08T16:59:18.684 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:194: set_size: local dir=td/test-erasure-eio 2024-05-08T16:59:18.684 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:195: set_size: shift 2024-05-08T16:59:18.684 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:196: set_size: local shard_id=1 2024-05-08T16:59:18.684 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:197: set_size: shift 2024-05-08T16:59:18.684 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:198: set_size: local bytes=10 2024-05-08T16:59:18.684 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:199: set_size: shift 2024-05-08T16:59:18.685 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:200: set_size: local mode=set 2024-05-08T16:59:18.685 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:202: set_size: local poolname=pool-jerasure 2024-05-08T16:59:18.685 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: get_osds pool-jerasure obj-size-81110-0-10 2024-05-08T16:59:18.686 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2024-05-08T16:59:18.686 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-size-81110-0-10 2024-05-08T16:59:18.686 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure obj-size-81110-0-10 2024-05-08T16:59:18.686 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-05-08T16:59:19.098 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-05-08T16:59:19.098 INFO:tasks.workunit.client.0.smithi172.stderr:1 2024-05-08T16:59:19.098 INFO:tasks.workunit.client.0.smithi172.stderr:2' 2024-05-08T16:59:19.098 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 2 2024-05-08T16:59:19.098 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: initial_osds=('3' '1' '2') 2024-05-08T16:59:19.098 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: local -a initial_osds 2024-05-08T16:59:19.099 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:204: set_size: local osd_id=1 2024-05-08T16:59:19.099 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:205: set_size: ceph osd set noout 2024-05-08T16:59:19.809 INFO:tasks.workunit.client.0.smithi172.stderr:noout is set 2024-05-08T16:59:19.832 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:206: set_size: '[' set = add ']' 2024-05-08T16:59:19.832 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:210: set_size: '[' 10 = 0 ']' 2024-05-08T16:59:19.832 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:214: set_size: dd if=/dev/urandom bs=10 count=1 of=td/test-erasure-eio/CORRUPT 2024-05-08T16:59:19.834 INFO:tasks.workunit.client.0.smithi172.stderr:1+0 records in 2024-05-08T16:59:19.834 INFO:tasks.workunit.client.0.smithi172.stderr:1+0 records out 2024-05-08T16:59:19.834 INFO:tasks.workunit.client.0.smithi172.stderr:10 bytes copied, 0.000116926 s, 85.5 kB/s 2024-05-08T16:59:19.834 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:216: set_size: objectstore_tool td/test-erasure-eio 1 obj-size-81110-0-10 set-bytes td/test-erasure-eio/CORRUPT 2024-05-08T16:59:19.834 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/test-erasure-eio 2024-05-08T16:59:19.834 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-05-08T16:59:19.834 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-05-08T16:59:19.835 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-05-08T16:59:19.835 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/test-erasure-eio 1 obj-size-81110-0-10 set-bytes td/test-erasure-eio/CORRUPT 2024-05-08T16:59:19.835 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/test-erasure-eio 2024-05-08T16:59:19.835 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-05-08T16:59:19.835 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-05-08T16:59:19.835 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-05-08T16:59:19.835 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/test-erasure-eio TERM osd.1 2024-05-08T16:59:19.836 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-08T16:59:19.836 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-08T16:59:19.836 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-08T16:59:19.836 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-08T16:59:19.836 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-08T16:59:21.147 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-08T16:59:21.148 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/test-erasure-eio 1 obj-size-81110-0-10 set-bytes td/test-erasure-eio/CORRUPT 2024-05-08T16:59:21.148 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/test-erasure-eio 2024-05-08T16:59:21.148 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-05-08T16:59:21.148 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-05-08T16:59:21.148 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-05-08T16:59:21.148 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/test-erasure-eio/1 2024-05-08T16:59:21.148 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/test-erasure-eio/1 obj-size-81110-0-10 set-bytes td/test-erasure-eio/CORRUPT 2024-05-08T16:59:23.260 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/test-erasure-eio 1 2024-05-08T16:59:23.260 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/test-erasure-eio 2024-05-08T16:59:23.260 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-05-08T16:59:23.260 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-05-08T16:59:23.260 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-05-08T16:59:23.261 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/test-erasure-eio/1 2024-05-08T16:59:23.261 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=4457b0f1-028b-482b-8ee3-7ee05861e6cc --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-05-08T16:59:23.261 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-08T16:59:23.261 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-05-08T16:59:23.261 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-08T16:59:23.261 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/1' 2024-05-08T16:59:23.261 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/1/journal' 2024-05-08T16:59:23.261 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-05-08T16:59:23.261 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-05-08T16:59:23.261 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-05-08T16:59:23.261 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-05-08T16:59:23.261 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-08T16:59:23.261 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-08T16:59:23.262 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-08T16:59:23.262 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T16:59:23.263 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81110 2024-05-08T16:59:23.263 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.81110/$cluster-$name.asok' 2024-05-08T16:59:23.263 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.81110/$cluster-$name.asok' 2024-05-08T16:59:23.263 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-05-08T16:59:23.263 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-05-08T16:59:23.263 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-05-08T16:59:23.263 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-08T16:59:23.263 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-08T16:59:23.263 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-08T16:59:23.263 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-08T16:59:23.263 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-05-08T16:59:23.263 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-05-08T16:59:23.263 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/test-erasure-eio/1 2024-05-08T16:59:23.264 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-05-08T16:59:23.264 INFO:tasks.workunit.client.0.smithi172.stderr:start osd.1 2024-05-08T16:59:23.264 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=4457b0f1-028b-482b-8ee3-7ee05861e6cc --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81110/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-05-08T16:59:23.264 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/test-erasure-eio/1/whoami 2024-05-08T16:59:23.265 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-05-08T16:59:23.265 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-05-08T16:59:23.266 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-05-08T16:59:23.266 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-05-08T16:59:23.298 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:59:23.301+0000 7fa0aeef17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:59:23.325 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:59:23.329+0000 7fa0aeef17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:59:23.327 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:59:23.329+0000 7fa0aeef17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:59:23.684 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-05-08T16:59:23.685 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-08T16:59:23.685 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-05-08T16:59:23.685 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-08T16:59:23.685 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-08T16:59:23.685 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:59:23.685 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-08T16:59:23.685 INFO:tasks.workunit.client.0.smithi172.stderr:0 2024-05-08T16:59:23.685 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:59:23.685 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T16:59:23.862 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:59:23.865+0000 7fa0aeef17c0 -1 Falling back to public interface 2024-05-08T16:59:24.114 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:59:25.116 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:59:25.116 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:59:25.116 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-08T16:59:25.116 INFO:tasks.workunit.client.0.smithi172.stderr:1 2024-05-08T16:59:25.117 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:59:25.117 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T16:59:25.558 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:59:26.505 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:59:26.509+0000 7fa0aeef17c0 -1 osd.1 48 log_to_monitors true 2024-05-08T16:59:26.560 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:59:26.560 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:59:26.560 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-08T16:59:26.560 INFO:tasks.workunit.client.0.smithi172.stderr:2 2024-05-08T16:59:26.560 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:59:26.560 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T16:59:27.232 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:59:28.234 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:59:28.234 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:59:28.234 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-05-08T16:59:28.234 INFO:tasks.workunit.client.0.smithi172.stderr:3 2024-05-08T16:59:28.234 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:59:28.234 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T16:59:28.810 INFO:tasks.workunit.client.0.smithi172.stderr:osd.1 up in weight 1 up_from 52 up_thru 52 down_at 49 last_clean_interval [13,48) [v2:127.0.0.1:6810/2329151812,v1:127.0.0.1:6811/2329151812] [v2:127.0.0.1:6812/2329151812,v1:127.0.0.1:6813/2329151812] exists,up 0f51dcbd-8561-4314-b628-17fe241f145f 2024-05-08T16:59:28.810 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-08T16:59:28.810 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-08T16:59:28.810 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-08T16:59:28.810 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-05-08T16:59:28.810 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-05-08T16:59:28.810 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-05-08T16:59:28.810 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-05-08T16:59:28.810 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-05-08T16:59:28.811 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-05-08T16:59:28.811 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-05-08T16:59:28.811 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-05-08T16:59:28.811 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-05-08T16:59:28.811 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-05-08T16:59:28.912 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-05-08T16:59:28.912 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-05-08T16:59:28.912 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-05-08T16:59:28.912 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-05-08T16:59:28.912 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-05-08T16:59:28.912 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-05-08T16:59:29.403 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-05-08T16:59:29.403 INFO:tasks.workunit.client.0.smithi172.stderr:1 2024-05-08T16:59:29.403 INFO:tasks.workunit.client.0.smithi172.stderr:2 2024-05-08T16:59:29.403 INFO:tasks.workunit.client.0.smithi172.stderr:3' 2024-05-08T16:59:29.404 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-05-08T16:59:29.404 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T16:59:29.404 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-05-08T16:59:29.582 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=25769803818 2024-05-08T16:59:29.582 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 25769803818 2024-05-08T16:59:29.582 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803818' 2024-05-08T16:59:29.583 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T16:59:29.583 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-05-08T16:59:29.732 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=223338299395 2024-05-08T16:59:29.732 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 223338299395 2024-05-08T16:59:29.732 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803818 1-223338299395' 2024-05-08T16:59:29.732 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T16:59:29.733 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-05-08T16:59:29.883 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=85899345947 2024-05-08T16:59:29.883 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 85899345947 2024-05-08T16:59:29.883 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803818 1-223338299395 2-85899345947' 2024-05-08T16:59:29.883 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T16:59:29.883 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-05-08T16:59:30.036 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=180388626440 2024-05-08T16:59:30.036 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 180388626440 2024-05-08T16:59:30.036 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803818 1-223338299395 2-85899345947 3-180388626440' 2024-05-08T16:59:30.036 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-08T16:59:30.036 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-25769803818 2024-05-08T16:59:30.037 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-08T16:59:30.037 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-05-08T16:59:30.038 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-25769803818 2024-05-08T16:59:30.038 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-08T16:59:30.039 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=25769803818 2024-05-08T16:59:30.039 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 25769803818' 2024-05-08T16:59:30.039 INFO:tasks.workunit.client.0.smithi172.stderr:waiting osd.0 seq 25769803818 2024-05-08T16:59:30.040 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-05-08T16:59:30.431 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 25769803816 -lt 25769803818 2024-05-08T16:59:30.431 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-05-08T16:59:31.432 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-05-08T16:59:31.433 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-05-08T16:59:31.852 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 25769803819 -lt 25769803818 2024-05-08T16:59:31.853 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-08T16:59:31.853 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-223338299395 2024-05-08T16:59:31.853 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-08T16:59:31.854 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-05-08T16:59:31.855 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-08T16:59:31.855 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-223338299395 2024-05-08T16:59:31.856 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=223338299395 2024-05-08T16:59:31.857 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 223338299395' 2024-05-08T16:59:31.857 INFO:tasks.workunit.client.0.smithi172.stderr:waiting osd.1 seq 223338299395 2024-05-08T16:59:31.857 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-05-08T16:59:32.364 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 223338299395 -lt 223338299395 2024-05-08T16:59:32.364 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-08T16:59:32.365 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-85899345947 2024-05-08T16:59:32.365 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-08T16:59:32.366 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-05-08T16:59:32.367 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-85899345947 2024-05-08T16:59:32.368 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-08T16:59:32.369 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=85899345947 2024-05-08T16:59:32.369 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 85899345947' 2024-05-08T16:59:32.369 INFO:tasks.workunit.client.0.smithi172.stderr:waiting osd.2 seq 85899345947 2024-05-08T16:59:32.369 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-05-08T16:59:32.791 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 85899345947 -lt 85899345947 2024-05-08T16:59:32.791 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-08T16:59:32.792 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-180388626440 2024-05-08T16:59:32.792 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-08T16:59:32.793 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-05-08T16:59:32.794 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-180388626440 2024-05-08T16:59:32.794 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-08T16:59:32.796 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=180388626440 2024-05-08T16:59:32.796 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 180388626440' 2024-05-08T16:59:32.796 INFO:tasks.workunit.client.0.smithi172.stderr:waiting osd.3 seq 180388626440 2024-05-08T16:59:32.796 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-05-08T16:59:33.216 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 180388626441 -lt 180388626440 2024-05-08T16:59:33.217 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-05-08T16:59:33.217 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-05-08T16:59:33.217 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-05-08T16:59:33.783 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-05-08T16:59:33.783 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-05-08T16:59:33.784 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-05-08T16:59:33.784 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-05-08T16:59:33.784 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-05-08T16:59:33.784 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-05-08T16:59:33.784 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-05-08T16:59:33.785 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-05-08T16:59:34.214 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-05-08T16:59:34.215 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-05-08T16:59:34.215 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-05-08T16:59:34.215 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-05-08T16:59:34.807 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-05-08T16:59:34.808 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-05-08T16:59:34.808 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-05-08T16:59:34.808 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:217: set_size: rm -f td/test-erasure-eio/CORRUPT 2024-05-08T16:59:34.808 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:218: set_size: ceph osd unset noout 2024-05-08T16:59:35.448 INFO:tasks.workunit.client.0.smithi172.stderr:noout is unset 2024-05-08T16:59:35.466 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:243: rados_get_data_bad_size: rados_get td/test-erasure-eio pool-jerasure obj-size-81110-0-10 fail 2024-05-08T16:59:35.466 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2024-05-08T16:59:35.466 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2024-05-08T16:59:35.466 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=obj-size-81110-0-10 2024-05-08T16:59:35.466 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=fail 2024-05-08T16:59:35.466 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' fail = fail ']' 2024-05-08T16:59:35.466 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:114: rados_get: rados --pool pool-jerasure get obj-size-81110-0-10 td/test-erasure-eio/COPY 2024-05-08T16:59:35.652 INFO:tasks.workunit.client.0.smithi172.stderr:error getting pool-jerasure/obj-size-81110-0-10: (5) Input/output error 2024-05-08T16:59:35.655 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:115: rados_get: return 2024-05-08T16:59:35.656 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:244: rados_get_data_bad_size: rm td/test-erasure-eio/ORIGINAL 2024-05-08T16:59:35.659 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:310: TEST_rados_get_bad_size_shard_0: rados_get_data_bad_size td/test-erasure-eio 0 0 2024-05-08T16:59:35.659 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:222: rados_get_data_bad_size: local dir=td/test-erasure-eio 2024-05-08T16:59:35.659 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:223: rados_get_data_bad_size: shift 2024-05-08T16:59:35.660 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:224: rados_get_data_bad_size: local shard_id=0 2024-05-08T16:59:35.660 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:225: rados_get_data_bad_size: shift 2024-05-08T16:59:35.660 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:226: rados_get_data_bad_size: local bytes=0 2024-05-08T16:59:35.660 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:227: rados_get_data_bad_size: shift 2024-05-08T16:59:35.660 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:228: rados_get_data_bad_size: local mode=set 2024-05-08T16:59:35.660 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:230: rados_get_data_bad_size: local poolname=pool-jerasure 2024-05-08T16:59:35.660 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:231: rados_get_data_bad_size: local objname=obj-size-81110-0-0 2024-05-08T16:59:35.660 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:232: rados_get_data_bad_size: rados_put td/test-erasure-eio pool-jerasure obj-size-81110-0-0 2024-05-08T16:59:35.660 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:90: rados_put: local dir=td/test-erasure-eio 2024-05-08T16:59:35.660 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:91: rados_put: local poolname=pool-jerasure 2024-05-08T16:59:35.660 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:92: rados_put: local objname=obj-size-81110-0-0 2024-05-08T16:59:35.661 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-05-08T16:59:35.661 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 AAA 2024-05-08T16:59:35.661 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-05-08T16:59:35.661 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 BBB 2024-05-08T16:59:35.661 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-05-08T16:59:35.661 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 CCCC 2024-05-08T16:59:35.661 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-05-08T16:59:35.661 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 DDDD 2024-05-08T16:59:35.661 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:100: rados_put: rados --pool pool-jerasure put obj-size-81110-0-0 td/test-erasure-eio/ORIGINAL 2024-05-08T16:59:35.832 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:236: rados_get_data_bad_size: set_size obj-size-81110-0-0 td/test-erasure-eio 0 0 set 2024-05-08T16:59:35.832 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:192: set_size: local objname=obj-size-81110-0-0 2024-05-08T16:59:35.832 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:193: set_size: shift 2024-05-08T16:59:35.833 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:194: set_size: local dir=td/test-erasure-eio 2024-05-08T16:59:35.833 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:195: set_size: shift 2024-05-08T16:59:35.833 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:196: set_size: local shard_id=0 2024-05-08T16:59:35.833 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:197: set_size: shift 2024-05-08T16:59:35.833 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:198: set_size: local bytes=0 2024-05-08T16:59:35.833 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:199: set_size: shift 2024-05-08T16:59:35.833 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:200: set_size: local mode=set 2024-05-08T16:59:35.833 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:202: set_size: local poolname=pool-jerasure 2024-05-08T16:59:35.833 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: get_osds pool-jerasure obj-size-81110-0-0 2024-05-08T16:59:35.833 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2024-05-08T16:59:35.833 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-size-81110-0-0 2024-05-08T16:59:35.834 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure obj-size-81110-0-0 2024-05-08T16:59:35.834 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-05-08T16:59:36.273 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-05-08T16:59:36.274 INFO:tasks.workunit.client.0.smithi172.stderr:1 2024-05-08T16:59:36.274 INFO:tasks.workunit.client.0.smithi172.stderr:2' 2024-05-08T16:59:36.274 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 2 2024-05-08T16:59:36.274 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: initial_osds=('3' '1' '2') 2024-05-08T16:59:36.274 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: local -a initial_osds 2024-05-08T16:59:36.274 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:204: set_size: local osd_id=3 2024-05-08T16:59:36.274 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:205: set_size: ceph osd set noout 2024-05-08T16:59:37.038 INFO:tasks.workunit.client.0.smithi172.stderr:noout is set 2024-05-08T16:59:37.059 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:206: set_size: '[' set = add ']' 2024-05-08T16:59:37.059 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:210: set_size: '[' 0 = 0 ']' 2024-05-08T16:59:37.059 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:212: set_size: touch td/test-erasure-eio/CORRUPT 2024-05-08T16:59:37.061 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:216: set_size: objectstore_tool td/test-erasure-eio 3 obj-size-81110-0-0 set-bytes td/test-erasure-eio/CORRUPT 2024-05-08T16:59:37.062 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/test-erasure-eio 2024-05-08T16:59:37.062 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-05-08T16:59:37.062 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=3 2024-05-08T16:59:37.062 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-05-08T16:59:37.062 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/test-erasure-eio 3 obj-size-81110-0-0 set-bytes td/test-erasure-eio/CORRUPT 2024-05-08T16:59:37.062 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/test-erasure-eio 2024-05-08T16:59:37.062 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-05-08T16:59:37.062 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=3 2024-05-08T16:59:37.062 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-05-08T16:59:37.062 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/test-erasure-eio TERM osd.3 2024-05-08T16:59:37.062 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-08T16:59:37.062 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-08T16:59:37.063 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-08T16:59:37.063 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-08T16:59:37.063 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-08T16:59:38.374 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-08T16:59:38.374 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/test-erasure-eio 3 obj-size-81110-0-0 set-bytes td/test-erasure-eio/CORRUPT 2024-05-08T16:59:38.374 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/test-erasure-eio 2024-05-08T16:59:38.375 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-05-08T16:59:38.375 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=3 2024-05-08T16:59:38.375 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-05-08T16:59:38.375 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/test-erasure-eio/3 2024-05-08T16:59:38.375 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/test-erasure-eio/3 obj-size-81110-0-0 set-bytes td/test-erasure-eio/CORRUPT 2024-05-08T16:59:40.247 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/test-erasure-eio 3 2024-05-08T16:59:40.247 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/test-erasure-eio 2024-05-08T16:59:40.247 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-05-08T16:59:40.247 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2024-05-08T16:59:40.248 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-05-08T16:59:40.248 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/test-erasure-eio/3 2024-05-08T16:59:40.248 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=4457b0f1-028b-482b-8ee3-7ee05861e6cc --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-05-08T16:59:40.248 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-08T16:59:40.248 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-05-08T16:59:40.248 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-08T16:59:40.248 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/3' 2024-05-08T16:59:40.248 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/3/journal' 2024-05-08T16:59:40.248 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-05-08T16:59:40.248 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-05-08T16:59:40.248 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-05-08T16:59:40.249 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-05-08T16:59:40.249 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-08T16:59:40.249 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-08T16:59:40.249 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-08T16:59:40.249 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T16:59:40.250 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81110 2024-05-08T16:59:40.251 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.81110/$cluster-$name.asok' 2024-05-08T16:59:40.251 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.81110/$cluster-$name.asok' 2024-05-08T16:59:40.251 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-05-08T16:59:40.251 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-05-08T16:59:40.251 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-05-08T16:59:40.251 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-08T16:59:40.251 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-08T16:59:40.252 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-08T16:59:40.252 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-08T16:59:40.252 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-05-08T16:59:40.252 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-05-08T16:59:40.252 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/test-erasure-eio/3 2024-05-08T16:59:40.252 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2024-05-08T16:59:40.252 INFO:tasks.workunit.client.0.smithi172.stderr:start osd.3 2024-05-08T16:59:40.253 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=4457b0f1-028b-482b-8ee3-7ee05861e6cc --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81110/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-05-08T16:59:40.253 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/test-erasure-eio/3/whoami 2024-05-08T16:59:40.254 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2024-05-08T16:59:40.255 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-05-08T16:59:40.255 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-05-08T16:59:40.256 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-05-08T16:59:40.286 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:59:40.288+0000 7f928f8687c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:59:40.289 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:59:40.292+0000 7f928f8687c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:59:40.291 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:59:40.292+0000 7f928f8687c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:59:40.668 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2024-05-08T16:59:40.668 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-08T16:59:40.668 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-05-08T16:59:40.668 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-08T16:59:40.668 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-08T16:59:40.668 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:59:40.669 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-08T16:59:40.669 INFO:tasks.workunit.client.0.smithi172.stderr:0 2024-05-08T16:59:40.669 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:59:40.669 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T16:59:40.833 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:59:40.836+0000 7f928f8687c0 -1 Falling back to public interface 2024-05-08T16:59:41.147 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:59:42.149 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:59:42.149 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:59:42.149 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-08T16:59:42.149 INFO:tasks.workunit.client.0.smithi172.stderr:1 2024-05-08T16:59:42.150 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:59:42.150 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T16:59:42.570 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:59:43.572 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:59:43.572 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:59:43.572 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-08T16:59:43.572 INFO:tasks.workunit.client.0.smithi172.stderr:2 2024-05-08T16:59:43.573 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:59:43.573 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T16:59:43.606 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:59:43.608+0000 7f928f8687c0 -1 osd.3 58 log_to_monitors true 2024-05-08T16:59:44.069 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:59:45.071 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:59:45.071 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:59:45.071 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-05-08T16:59:45.071 INFO:tasks.workunit.client.0.smithi172.stderr:3 2024-05-08T16:59:45.072 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:59:45.072 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T16:59:45.758 INFO:tasks.workunit.client.0.smithi172.stderr:osd.3 up in weight 1 up_from 62 up_thru 62 down_at 59 last_clean_interval [42,58) [v2:127.0.0.1:6826/2257747150,v1:127.0.0.1:6827/2257747150] [v2:127.0.0.1:6828/2257747150,v1:127.0.0.1:6829/2257747150] exists,up 724678d7-ca35-4cdd-a1aa-910a2045b6f4 2024-05-08T16:59:45.758 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-08T16:59:45.758 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-08T16:59:45.758 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-08T16:59:45.759 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-05-08T16:59:45.759 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-05-08T16:59:45.759 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-05-08T16:59:45.759 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-05-08T16:59:45.759 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-05-08T16:59:45.759 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-05-08T16:59:45.760 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-05-08T16:59:45.760 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-05-08T16:59:45.760 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-05-08T16:59:45.760 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-05-08T16:59:45.874 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-05-08T16:59:45.874 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-05-08T16:59:45.874 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-05-08T16:59:45.874 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-05-08T16:59:45.874 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-05-08T16:59:45.874 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-05-08T16:59:46.295 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-05-08T16:59:46.295 INFO:tasks.workunit.client.0.smithi172.stderr:1 2024-05-08T16:59:46.296 INFO:tasks.workunit.client.0.smithi172.stderr:2 2024-05-08T16:59:46.296 INFO:tasks.workunit.client.0.smithi172.stderr:3' 2024-05-08T16:59:46.296 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-05-08T16:59:46.296 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T16:59:46.296 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-05-08T16:59:46.472 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=25769803824 2024-05-08T16:59:46.473 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 25769803824 2024-05-08T16:59:46.473 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803824' 2024-05-08T16:59:46.473 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T16:59:46.473 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-05-08T16:59:46.617 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=223338299400 2024-05-08T16:59:46.617 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 223338299400 2024-05-08T16:59:46.617 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803824 1-223338299400' 2024-05-08T16:59:46.617 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T16:59:46.617 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-05-08T16:59:46.761 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=85899345952 2024-05-08T16:59:46.762 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 85899345952 2024-05-08T16:59:46.762 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803824 1-223338299400 2-85899345952' 2024-05-08T16:59:46.762 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T16:59:46.762 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-05-08T16:59:46.919 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=266287972355 2024-05-08T16:59:46.919 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 266287972355 2024-05-08T16:59:46.919 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803824 1-223338299400 2-85899345952 3-266287972355' 2024-05-08T16:59:46.919 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-08T16:59:46.920 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-25769803824 2024-05-08T16:59:46.920 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-08T16:59:46.921 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-05-08T16:59:46.922 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-25769803824 2024-05-08T16:59:46.922 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-08T16:59:46.923 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=25769803824 2024-05-08T16:59:46.923 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 25769803824' 2024-05-08T16:59:46.923 INFO:tasks.workunit.client.0.smithi172.stderr:waiting osd.0 seq 25769803824 2024-05-08T16:59:46.923 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-05-08T16:59:47.358 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 25769803824 -lt 25769803824 2024-05-08T16:59:47.359 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-08T16:59:47.359 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-223338299400 2024-05-08T16:59:47.360 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-08T16:59:47.361 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-05-08T16:59:47.362 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-223338299400 2024-05-08T16:59:47.362 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-08T16:59:47.363 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=223338299400 2024-05-08T16:59:47.363 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 223338299400' 2024-05-08T16:59:47.364 INFO:tasks.workunit.client.0.smithi172.stderr:waiting osd.1 seq 223338299400 2024-05-08T16:59:47.364 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-05-08T16:59:47.858 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 223338299400 -lt 223338299400 2024-05-08T16:59:47.858 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-08T16:59:47.859 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-85899345952 2024-05-08T16:59:47.859 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-08T16:59:47.861 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-05-08T16:59:47.862 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-85899345952 2024-05-08T16:59:47.862 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-08T16:59:47.863 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=85899345952 2024-05-08T16:59:47.863 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 85899345952' 2024-05-08T16:59:47.863 INFO:tasks.workunit.client.0.smithi172.stderr:waiting osd.2 seq 85899345952 2024-05-08T16:59:47.864 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-05-08T16:59:48.303 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 85899345951 -lt 85899345952 2024-05-08T16:59:48.303 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-05-08T16:59:49.305 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-05-08T16:59:49.306 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-05-08T16:59:49.709 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 85899345953 -lt 85899345952 2024-05-08T16:59:49.709 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-08T16:59:49.710 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-08T16:59:49.710 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-266287972355 2024-05-08T16:59:49.711 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-05-08T16:59:49.712 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-266287972355 2024-05-08T16:59:49.712 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-08T16:59:49.713 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=266287972355 2024-05-08T16:59:49.713 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 266287972355' 2024-05-08T16:59:49.713 INFO:tasks.workunit.client.0.smithi172.stderr:waiting osd.3 seq 266287972355 2024-05-08T16:59:49.713 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-05-08T16:59:50.148 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 266287972355 -lt 266287972355 2024-05-08T16:59:50.148 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-05-08T16:59:50.149 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-05-08T16:59:50.149 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-05-08T16:59:50.680 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-05-08T16:59:50.680 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-05-08T16:59:50.681 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-05-08T16:59:50.681 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-05-08T16:59:50.681 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-05-08T16:59:50.681 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-05-08T16:59:50.681 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-05-08T16:59:50.681 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-05-08T16:59:51.135 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-05-08T16:59:51.136 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-05-08T16:59:51.136 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-05-08T16:59:51.136 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-05-08T16:59:51.676 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-05-08T16:59:51.676 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-05-08T16:59:51.677 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-05-08T16:59:51.677 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:217: set_size: rm -f td/test-erasure-eio/CORRUPT 2024-05-08T16:59:51.677 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:218: set_size: ceph osd unset noout 2024-05-08T16:59:52.316 INFO:tasks.workunit.client.0.smithi172.stderr:noout is unset 2024-05-08T16:59:52.339 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:238: rados_get_data_bad_size: rados_get td/test-erasure-eio pool-jerasure obj-size-81110-0-0 2024-05-08T16:59:52.339 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2024-05-08T16:59:52.339 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2024-05-08T16:59:52.339 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=obj-size-81110-0-0 2024-05-08T16:59:52.339 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=ok 2024-05-08T16:59:52.339 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' ok = fail ']' 2024-05-08T16:59:52.339 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:120: rados_get: rados --pool pool-jerasure get obj-size-81110-0-0 td/test-erasure-eio/COPY 2024-05-08T16:59:52.556 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:121: rados_get: diff td/test-erasure-eio/ORIGINAL td/test-erasure-eio/COPY 2024-05-08T16:59:52.558 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:122: rados_get: rm td/test-erasure-eio/COPY 2024-05-08T16:59:52.568 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:241: rados_get_data_bad_size: expr 0 + 1 2024-05-08T16:59:52.570 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:241: rados_get_data_bad_size: shard_id=1 2024-05-08T16:59:52.570 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:242: rados_get_data_bad_size: set_size obj-size-81110-0-0 td/test-erasure-eio 1 0 set 2024-05-08T16:59:52.570 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:192: set_size: local objname=obj-size-81110-0-0 2024-05-08T16:59:52.570 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:193: set_size: shift 2024-05-08T16:59:52.570 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:194: set_size: local dir=td/test-erasure-eio 2024-05-08T16:59:52.571 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:195: set_size: shift 2024-05-08T16:59:52.571 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:196: set_size: local shard_id=1 2024-05-08T16:59:52.571 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:197: set_size: shift 2024-05-08T16:59:52.571 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:198: set_size: local bytes=0 2024-05-08T16:59:52.571 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:199: set_size: shift 2024-05-08T16:59:52.571 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:200: set_size: local mode=set 2024-05-08T16:59:52.571 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:202: set_size: local poolname=pool-jerasure 2024-05-08T16:59:52.572 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: get_osds pool-jerasure obj-size-81110-0-0 2024-05-08T16:59:52.572 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2024-05-08T16:59:52.572 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-size-81110-0-0 2024-05-08T16:59:52.572 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure obj-size-81110-0-0 2024-05-08T16:59:52.572 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-05-08T16:59:53.030 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-05-08T16:59:53.030 INFO:tasks.workunit.client.0.smithi172.stderr:1 2024-05-08T16:59:53.030 INFO:tasks.workunit.client.0.smithi172.stderr:2' 2024-05-08T16:59:53.030 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 2 2024-05-08T16:59:53.030 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: initial_osds=('3' '1' '2') 2024-05-08T16:59:53.030 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: local -a initial_osds 2024-05-08T16:59:53.030 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:204: set_size: local osd_id=1 2024-05-08T16:59:53.030 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:205: set_size: ceph osd set noout 2024-05-08T16:59:53.856 INFO:tasks.workunit.client.0.smithi172.stderr:noout is set 2024-05-08T16:59:53.876 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:206: set_size: '[' set = add ']' 2024-05-08T16:59:53.876 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:210: set_size: '[' 0 = 0 ']' 2024-05-08T16:59:53.876 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:212: set_size: touch td/test-erasure-eio/CORRUPT 2024-05-08T16:59:53.879 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:216: set_size: objectstore_tool td/test-erasure-eio 1 obj-size-81110-0-0 set-bytes td/test-erasure-eio/CORRUPT 2024-05-08T16:59:53.879 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/test-erasure-eio 2024-05-08T16:59:53.879 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-05-08T16:59:53.879 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-05-08T16:59:53.879 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-05-08T16:59:53.879 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/test-erasure-eio 1 obj-size-81110-0-0 set-bytes td/test-erasure-eio/CORRUPT 2024-05-08T16:59:53.879 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/test-erasure-eio 2024-05-08T16:59:53.879 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-05-08T16:59:53.879 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-05-08T16:59:53.879 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-05-08T16:59:53.880 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/test-erasure-eio TERM osd.1 2024-05-08T16:59:53.880 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-08T16:59:53.880 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-08T16:59:53.880 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-08T16:59:53.880 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-08T16:59:53.880 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-08T16:59:55.192 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-08T16:59:55.192 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/test-erasure-eio 1 obj-size-81110-0-0 set-bytes td/test-erasure-eio/CORRUPT 2024-05-08T16:59:55.192 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/test-erasure-eio 2024-05-08T16:59:55.192 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-05-08T16:59:55.192 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-05-08T16:59:55.192 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-05-08T16:59:55.192 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/test-erasure-eio/1 2024-05-08T16:59:55.192 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/test-erasure-eio/1 obj-size-81110-0-0 set-bytes td/test-erasure-eio/CORRUPT 2024-05-08T16:59:56.973 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/test-erasure-eio 1 2024-05-08T16:59:56.973 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/test-erasure-eio 2024-05-08T16:59:56.973 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-05-08T16:59:56.973 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-05-08T16:59:56.974 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-05-08T16:59:56.974 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/test-erasure-eio/1 2024-05-08T16:59:56.974 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=4457b0f1-028b-482b-8ee3-7ee05861e6cc --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-05-08T16:59:56.974 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-08T16:59:56.974 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-05-08T16:59:56.974 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-08T16:59:56.974 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/1' 2024-05-08T16:59:56.974 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/1/journal' 2024-05-08T16:59:56.974 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-05-08T16:59:56.974 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-05-08T16:59:56.974 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-05-08T16:59:56.974 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-05-08T16:59:56.974 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-08T16:59:56.974 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-08T16:59:56.975 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-08T16:59:56.975 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T16:59:56.975 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81110 2024-05-08T16:59:56.975 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.81110/$cluster-$name.asok' 2024-05-08T16:59:56.976 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.81110/$cluster-$name.asok' 2024-05-08T16:59:56.976 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-05-08T16:59:56.976 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-05-08T16:59:56.976 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-05-08T16:59:56.976 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-08T16:59:56.976 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-08T16:59:56.976 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-08T16:59:56.976 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-08T16:59:56.976 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-05-08T16:59:56.976 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-05-08T16:59:56.977 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/test-erasure-eio/1 2024-05-08T16:59:56.977 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-05-08T16:59:56.977 INFO:tasks.workunit.client.0.smithi172.stderr:start osd.1 2024-05-08T16:59:56.977 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=4457b0f1-028b-482b-8ee3-7ee05861e6cc --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81110/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-05-08T16:59:56.977 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/test-erasure-eio/1/whoami 2024-05-08T16:59:56.978 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-05-08T16:59:56.979 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-05-08T16:59:56.979 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-05-08T16:59:56.979 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-05-08T16:59:57.009 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:59:57.012+0000 7f0b7df847c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:59:57.012 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:59:57.016+0000 7f0b7df847c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:59:57.013 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:59:57.016+0000 7f0b7df847c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T16:59:57.399 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-05-08T16:59:57.400 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-08T16:59:57.400 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-05-08T16:59:57.400 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-08T16:59:57.400 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-08T16:59:57.400 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:59:57.400 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-08T16:59:57.400 INFO:tasks.workunit.client.0.smithi172.stderr:0 2024-05-08T16:59:57.400 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:59:57.400 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T16:59:57.570 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T16:59:57.572+0000 7f0b7df847c0 -1 Falling back to public interface 2024-05-08T16:59:57.834 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T16:59:58.836 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T16:59:58.836 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T16:59:58.836 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-08T16:59:58.836 INFO:tasks.workunit.client.0.smithi172.stderr:1 2024-05-08T16:59:58.837 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T16:59:58.837 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T16:59:59.238 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:00:00.197 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:00:00.200+0000 7f0b7df847c0 -1 osd.1 67 log_to_monitors true 2024-05-08T17:00:00.240 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:00:00.240 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:00:00.240 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-08T17:00:00.240 INFO:tasks.workunit.client.0.smithi172.stderr:2 2024-05-08T17:00:00.241 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:00:00.241 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T17:00:00.758 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:00:01.761 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:00:01.761 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:00:01.761 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-05-08T17:00:01.761 INFO:tasks.workunit.client.0.smithi172.stderr:3 2024-05-08T17:00:01.761 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:00:01.761 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T17:00:02.285 INFO:tasks.workunit.client.0.smithi172.stderr:osd.1 up in weight 1 up_from 72 up_thru 72 down_at 69 last_clean_interval [52,67) [v2:127.0.0.1:6810/3977503487,v1:127.0.0.1:6811/3977503487] [v2:127.0.0.1:6812/3977503487,v1:127.0.0.1:6813/3977503487] exists,up 0f51dcbd-8561-4314-b628-17fe241f145f 2024-05-08T17:00:02.286 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-08T17:00:02.286 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-08T17:00:02.286 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-08T17:00:02.286 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-05-08T17:00:02.286 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-05-08T17:00:02.286 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-05-08T17:00:02.286 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-05-08T17:00:02.287 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-05-08T17:00:02.287 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-05-08T17:00:02.287 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-05-08T17:00:02.288 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-05-08T17:00:02.288 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-05-08T17:00:02.288 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-05-08T17:00:02.399 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-05-08T17:00:02.399 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-05-08T17:00:02.399 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-05-08T17:00:02.399 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-05-08T17:00:02.399 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-05-08T17:00:02.400 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-05-08T17:00:02.845 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-05-08T17:00:02.845 INFO:tasks.workunit.client.0.smithi172.stderr:1 2024-05-08T17:00:02.845 INFO:tasks.workunit.client.0.smithi172.stderr:2 2024-05-08T17:00:02.845 INFO:tasks.workunit.client.0.smithi172.stderr:3' 2024-05-08T17:00:02.845 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-05-08T17:00:02.845 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T17:00:02.846 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-05-08T17:00:03.002 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=25769803829 2024-05-08T17:00:03.002 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 25769803829 2024-05-08T17:00:03.002 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803829' 2024-05-08T17:00:03.002 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T17:00:03.002 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-05-08T17:00:03.151 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=309237645315 2024-05-08T17:00:03.152 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 309237645315 2024-05-08T17:00:03.152 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803829 1-309237645315' 2024-05-08T17:00:03.152 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T17:00:03.152 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-05-08T17:00:03.306 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=85899345957 2024-05-08T17:00:03.306 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 85899345957 2024-05-08T17:00:03.306 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803829 1-309237645315 2-85899345957' 2024-05-08T17:00:03.306 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T17:00:03.306 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-05-08T17:00:03.450 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=266287972360 2024-05-08T17:00:03.450 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 266287972360 2024-05-08T17:00:03.450 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803829 1-309237645315 2-85899345957 3-266287972360' 2024-05-08T17:00:03.450 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-08T17:00:03.451 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-25769803829 2024-05-08T17:00:03.451 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-08T17:00:03.451 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-05-08T17:00:03.452 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-25769803829 2024-05-08T17:00:03.452 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-08T17:00:03.453 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=25769803829 2024-05-08T17:00:03.453 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 25769803829' 2024-05-08T17:00:03.454 INFO:tasks.workunit.client.0.smithi172.stderr:waiting osd.0 seq 25769803829 2024-05-08T17:00:03.454 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-05-08T17:00:03.846 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 25769803827 -lt 25769803829 2024-05-08T17:00:03.846 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-05-08T17:00:04.848 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-05-08T17:00:04.849 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-05-08T17:00:05.299 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 25769803829 -lt 25769803829 2024-05-08T17:00:05.300 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-08T17:00:05.300 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-309237645315 2024-05-08T17:00:05.301 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-08T17:00:05.302 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-05-08T17:00:05.303 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-309237645315 2024-05-08T17:00:05.303 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-08T17:00:05.304 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=309237645315 2024-05-08T17:00:05.304 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 309237645315' 2024-05-08T17:00:05.304 INFO:tasks.workunit.client.0.smithi172.stderr:waiting osd.1 seq 309237645315 2024-05-08T17:00:05.305 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-05-08T17:00:05.754 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 309237645315 -lt 309237645315 2024-05-08T17:00:05.754 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-08T17:00:05.755 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-08T17:00:05.755 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-85899345957 2024-05-08T17:00:05.756 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-05-08T17:00:05.757 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-08T17:00:05.757 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-85899345957 2024-05-08T17:00:05.759 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=85899345957 2024-05-08T17:00:05.759 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 85899345957' 2024-05-08T17:00:05.760 INFO:tasks.workunit.client.0.smithi172.stderr:waiting osd.2 seq 85899345957 2024-05-08T17:00:05.760 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-05-08T17:00:06.186 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 85899345958 -lt 85899345957 2024-05-08T17:00:06.186 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-08T17:00:06.187 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-08T17:00:06.187 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-266287972360 2024-05-08T17:00:06.188 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-05-08T17:00:06.189 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-266287972360 2024-05-08T17:00:06.189 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-08T17:00:06.191 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=266287972360 2024-05-08T17:00:06.191 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 266287972360' 2024-05-08T17:00:06.191 INFO:tasks.workunit.client.0.smithi172.stderr:waiting osd.3 seq 266287972360 2024-05-08T17:00:06.192 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-05-08T17:00:06.611 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 266287972361 -lt 266287972360 2024-05-08T17:00:06.612 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-05-08T17:00:06.612 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-05-08T17:00:06.613 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-05-08T17:00:07.145 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-05-08T17:00:07.145 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-05-08T17:00:07.145 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-05-08T17:00:07.145 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-05-08T17:00:07.145 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-05-08T17:00:07.145 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-05-08T17:00:07.146 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-05-08T17:00:07.146 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-05-08T17:00:07.565 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-05-08T17:00:07.565 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-05-08T17:00:07.565 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-05-08T17:00:07.566 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-05-08T17:00:08.083 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-05-08T17:00:08.083 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-05-08T17:00:08.083 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-05-08T17:00:08.083 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:217: set_size: rm -f td/test-erasure-eio/CORRUPT 2024-05-08T17:00:08.084 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:218: set_size: ceph osd unset noout 2024-05-08T17:00:08.934 INFO:tasks.workunit.client.0.smithi172.stderr:noout is unset 2024-05-08T17:00:08.956 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:243: rados_get_data_bad_size: rados_get td/test-erasure-eio pool-jerasure obj-size-81110-0-0 fail 2024-05-08T17:00:08.956 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2024-05-08T17:00:08.956 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2024-05-08T17:00:08.957 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=obj-size-81110-0-0 2024-05-08T17:00:08.957 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=fail 2024-05-08T17:00:08.957 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' fail = fail ']' 2024-05-08T17:00:08.957 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:114: rados_get: rados --pool pool-jerasure get obj-size-81110-0-0 td/test-erasure-eio/COPY 2024-05-08T17:00:09.138 INFO:tasks.workunit.client.0.smithi172.stderr:error getting pool-jerasure/obj-size-81110-0-0: (5) Input/output error 2024-05-08T17:00:09.143 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:115: rados_get: return 2024-05-08T17:00:09.143 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:244: rados_get_data_bad_size: rm td/test-erasure-eio/ORIGINAL 2024-05-08T17:00:09.146 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:311: TEST_rados_get_bad_size_shard_0: rados_get_data_bad_size td/test-erasure-eio 0 256 add 2024-05-08T17:00:09.146 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:222: rados_get_data_bad_size: local dir=td/test-erasure-eio 2024-05-08T17:00:09.146 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:223: rados_get_data_bad_size: shift 2024-05-08T17:00:09.146 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:224: rados_get_data_bad_size: local shard_id=0 2024-05-08T17:00:09.147 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:225: rados_get_data_bad_size: shift 2024-05-08T17:00:09.147 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:226: rados_get_data_bad_size: local bytes=256 2024-05-08T17:00:09.147 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:227: rados_get_data_bad_size: shift 2024-05-08T17:00:09.147 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:228: rados_get_data_bad_size: local mode=add 2024-05-08T17:00:09.147 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:230: rados_get_data_bad_size: local poolname=pool-jerasure 2024-05-08T17:00:09.147 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:231: rados_get_data_bad_size: local objname=obj-size-81110-0-256 2024-05-08T17:00:09.147 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:232: rados_get_data_bad_size: rados_put td/test-erasure-eio pool-jerasure obj-size-81110-0-256 2024-05-08T17:00:09.147 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:90: rados_put: local dir=td/test-erasure-eio 2024-05-08T17:00:09.147 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:91: rados_put: local poolname=pool-jerasure 2024-05-08T17:00:09.147 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:92: rados_put: local objname=obj-size-81110-0-256 2024-05-08T17:00:09.147 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-05-08T17:00:09.147 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 AAA 2024-05-08T17:00:09.147 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-05-08T17:00:09.147 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 BBB 2024-05-08T17:00:09.147 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-05-08T17:00:09.147 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 CCCC 2024-05-08T17:00:09.148 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-05-08T17:00:09.148 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 DDDD 2024-05-08T17:00:09.148 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:100: rados_put: rados --pool pool-jerasure put obj-size-81110-0-256 td/test-erasure-eio/ORIGINAL 2024-05-08T17:00:09.343 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:236: rados_get_data_bad_size: set_size obj-size-81110-0-256 td/test-erasure-eio 0 256 add 2024-05-08T17:00:09.343 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:192: set_size: local objname=obj-size-81110-0-256 2024-05-08T17:00:09.344 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:193: set_size: shift 2024-05-08T17:00:09.344 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:194: set_size: local dir=td/test-erasure-eio 2024-05-08T17:00:09.344 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:195: set_size: shift 2024-05-08T17:00:09.344 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:196: set_size: local shard_id=0 2024-05-08T17:00:09.344 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:197: set_size: shift 2024-05-08T17:00:09.344 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:198: set_size: local bytes=256 2024-05-08T17:00:09.344 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:199: set_size: shift 2024-05-08T17:00:09.344 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:200: set_size: local mode=add 2024-05-08T17:00:09.344 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:202: set_size: local poolname=pool-jerasure 2024-05-08T17:00:09.344 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: get_osds pool-jerasure obj-size-81110-0-256 2024-05-08T17:00:09.344 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2024-05-08T17:00:09.344 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-size-81110-0-256 2024-05-08T17:00:09.345 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure obj-size-81110-0-256 2024-05-08T17:00:09.345 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-05-08T17:00:09.822 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-05-08T17:00:09.823 INFO:tasks.workunit.client.0.smithi172.stderr:1 2024-05-08T17:00:09.823 INFO:tasks.workunit.client.0.smithi172.stderr:2' 2024-05-08T17:00:09.823 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 2 2024-05-08T17:00:09.823 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: initial_osds=('3' '1' '2') 2024-05-08T17:00:09.823 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: local -a initial_osds 2024-05-08T17:00:09.823 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:204: set_size: local osd_id=3 2024-05-08T17:00:09.823 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:205: set_size: ceph osd set noout 2024-05-08T17:00:10.517 INFO:tasks.workunit.client.0.smithi172.stderr:noout is set 2024-05-08T17:00:10.538 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:206: set_size: '[' add = add ']' 2024-05-08T17:00:10.538 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:208: set_size: objectstore_tool td/test-erasure-eio 3 obj-size-81110-0-256 get-bytes td/test-erasure-eio/CORRUPT 2024-05-08T17:00:10.539 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/test-erasure-eio 2024-05-08T17:00:10.539 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-05-08T17:00:10.539 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=3 2024-05-08T17:00:10.539 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-05-08T17:00:10.539 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/test-erasure-eio 3 obj-size-81110-0-256 get-bytes td/test-erasure-eio/CORRUPT 2024-05-08T17:00:10.539 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/test-erasure-eio 2024-05-08T17:00:10.539 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-05-08T17:00:10.539 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=3 2024-05-08T17:00:10.539 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-05-08T17:00:10.539 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/test-erasure-eio TERM osd.3 2024-05-08T17:00:10.539 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-08T17:00:10.539 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-08T17:00:10.540 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-08T17:00:10.540 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-08T17:00:10.540 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-08T17:00:11.851 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-08T17:00:11.852 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/test-erasure-eio 3 obj-size-81110-0-256 get-bytes td/test-erasure-eio/CORRUPT 2024-05-08T17:00:11.852 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/test-erasure-eio 2024-05-08T17:00:11.852 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-05-08T17:00:11.852 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=3 2024-05-08T17:00:11.852 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-05-08T17:00:11.852 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/test-erasure-eio/3 2024-05-08T17:00:11.852 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/test-erasure-eio/3 obj-size-81110-0-256 get-bytes td/test-erasure-eio/CORRUPT 2024-05-08T17:00:13.636 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/test-erasure-eio 3 2024-05-08T17:00:13.636 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/test-erasure-eio 2024-05-08T17:00:13.636 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-05-08T17:00:13.636 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2024-05-08T17:00:13.636 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-05-08T17:00:13.636 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/test-erasure-eio/3 2024-05-08T17:00:13.636 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=4457b0f1-028b-482b-8ee3-7ee05861e6cc --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-05-08T17:00:13.636 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-08T17:00:13.636 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-05-08T17:00:13.636 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-08T17:00:13.636 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/3' 2024-05-08T17:00:13.637 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/3/journal' 2024-05-08T17:00:13.637 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-05-08T17:00:13.637 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-05-08T17:00:13.637 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-05-08T17:00:13.639 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-05-08T17:00:13.639 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-08T17:00:13.639 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-08T17:00:13.639 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-08T17:00:13.639 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T17:00:13.639 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81110 2024-05-08T17:00:13.639 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.81110/$cluster-$name.asok' 2024-05-08T17:00:13.639 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.81110/$cluster-$name.asok' 2024-05-08T17:00:13.639 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-05-08T17:00:13.639 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-05-08T17:00:13.639 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-05-08T17:00:13.639 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-08T17:00:13.639 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-08T17:00:13.640 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-08T17:00:13.640 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-08T17:00:13.640 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-05-08T17:00:13.640 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-05-08T17:00:13.640 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/test-erasure-eio/3 2024-05-08T17:00:13.640 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2024-05-08T17:00:13.640 INFO:tasks.workunit.client.0.smithi172.stderr:start osd.3 2024-05-08T17:00:13.641 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=4457b0f1-028b-482b-8ee3-7ee05861e6cc --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81110/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-05-08T17:00:13.642 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/test-erasure-eio/3/whoami 2024-05-08T17:00:13.642 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2024-05-08T17:00:13.643 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-05-08T17:00:13.643 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-05-08T17:00:13.643 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-05-08T17:00:13.674 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:00:13.676+0000 7ff1ffe4b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T17:00:13.677 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:00:13.680+0000 7ff1ffe4b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T17:00:13.678 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:00:13.680+0000 7ff1ffe4b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T17:00:14.072 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2024-05-08T17:00:14.072 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-08T17:00:14.072 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-05-08T17:00:14.072 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-08T17:00:14.072 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-08T17:00:14.072 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:00:14.073 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-08T17:00:14.073 INFO:tasks.workunit.client.0.smithi172.stderr:0 2024-05-08T17:00:14.073 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:00:14.073 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T17:00:14.209 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:00:14.212+0000 7ff1ffe4b7c0 -1 Falling back to public interface 2024-05-08T17:00:14.520 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:00:15.522 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:00:15.522 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:00:15.522 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-08T17:00:15.523 INFO:tasks.workunit.client.0.smithi172.stderr:1 2024-05-08T17:00:15.523 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:00:15.523 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T17:00:15.931 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:00:16.919 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:00:16.920+0000 7ff1ffe4b7c0 -1 osd.3 77 log_to_monitors true 2024-05-08T17:00:16.933 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:00:16.933 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:00:16.933 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-08T17:00:16.933 INFO:tasks.workunit.client.0.smithi172.stderr:2 2024-05-08T17:00:16.933 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:00:16.933 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T17:00:17.433 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:00:18.435 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:00:18.435 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:00:18.435 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-05-08T17:00:18.436 INFO:tasks.workunit.client.0.smithi172.stderr:3 2024-05-08T17:00:18.436 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:00:18.436 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T17:00:19.295 INFO:tasks.workunit.client.0.smithi172.stderr:osd.3 up in weight 1 up_from 82 up_thru 82 down_at 79 last_clean_interval [62,77) [v2:127.0.0.1:6826/2881341895,v1:127.0.0.1:6827/2881341895] [v2:127.0.0.1:6828/2881341895,v1:127.0.0.1:6829/2881341895] exists,up 724678d7-ca35-4cdd-a1aa-910a2045b6f4 2024-05-08T17:00:19.295 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-08T17:00:19.295 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-08T17:00:19.295 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-08T17:00:19.295 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-05-08T17:00:19.295 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-05-08T17:00:19.295 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-05-08T17:00:19.295 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-05-08T17:00:19.296 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-05-08T17:00:19.296 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-05-08T17:00:19.296 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-05-08T17:00:19.296 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-05-08T17:00:19.296 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-05-08T17:00:19.297 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-05-08T17:00:19.414 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-05-08T17:00:19.414 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-05-08T17:00:19.415 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-05-08T17:00:19.415 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-05-08T17:00:19.415 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-05-08T17:00:19.415 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-05-08T17:00:20.011 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-05-08T17:00:20.012 INFO:tasks.workunit.client.0.smithi172.stderr:1 2024-05-08T17:00:20.012 INFO:tasks.workunit.client.0.smithi172.stderr:2 2024-05-08T17:00:20.012 INFO:tasks.workunit.client.0.smithi172.stderr:3' 2024-05-08T17:00:20.012 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-05-08T17:00:20.012 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T17:00:20.012 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-05-08T17:00:20.179 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=25769803834 2024-05-08T17:00:20.179 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 25769803834 2024-05-08T17:00:20.179 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803834' 2024-05-08T17:00:20.179 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T17:00:20.180 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-05-08T17:00:20.326 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=309237645320 2024-05-08T17:00:20.326 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 309237645320 2024-05-08T17:00:20.327 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803834 1-309237645320' 2024-05-08T17:00:20.327 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T17:00:20.327 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-05-08T17:00:20.477 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=85899345963 2024-05-08T17:00:20.477 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 85899345963 2024-05-08T17:00:20.477 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803834 1-309237645320 2-85899345963' 2024-05-08T17:00:20.477 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T17:00:20.477 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-05-08T17:00:20.634 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=352187318275 2024-05-08T17:00:20.634 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 352187318275 2024-05-08T17:00:20.634 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803834 1-309237645320 2-85899345963 3-352187318275' 2024-05-08T17:00:20.634 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-08T17:00:20.635 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-08T17:00:20.635 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-25769803834 2024-05-08T17:00:20.636 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-05-08T17:00:20.637 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-25769803834 2024-05-08T17:00:20.637 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-08T17:00:20.638 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=25769803834 2024-05-08T17:00:20.638 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 25769803834' 2024-05-08T17:00:20.638 INFO:tasks.workunit.client.0.smithi172.stderr:waiting osd.0 seq 25769803834 2024-05-08T17:00:20.639 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-05-08T17:00:21.040 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 25769803835 -lt 25769803834 2024-05-08T17:00:21.041 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-08T17:00:21.041 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-309237645320 2024-05-08T17:00:21.042 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-08T17:00:21.043 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-05-08T17:00:21.044 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-309237645320 2024-05-08T17:00:21.044 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-08T17:00:21.046 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=309237645320 2024-05-08T17:00:21.046 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 309237645320' 2024-05-08T17:00:21.046 INFO:tasks.workunit.client.0.smithi172.stderr:waiting osd.1 seq 309237645320 2024-05-08T17:00:21.046 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-05-08T17:00:21.447 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 309237645321 -lt 309237645320 2024-05-08T17:00:21.447 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-08T17:00:21.447 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-85899345963 2024-05-08T17:00:21.448 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-08T17:00:21.449 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-05-08T17:00:21.450 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-85899345963 2024-05-08T17:00:21.450 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-08T17:00:21.451 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=85899345963 2024-05-08T17:00:21.451 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 85899345963' 2024-05-08T17:00:21.451 INFO:tasks.workunit.client.0.smithi172.stderr:waiting osd.2 seq 85899345963 2024-05-08T17:00:21.451 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-05-08T17:00:21.855 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 85899345963 -lt 85899345963 2024-05-08T17:00:21.855 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-08T17:00:21.856 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-352187318275 2024-05-08T17:00:21.856 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-08T17:00:21.857 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-05-08T17:00:21.858 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-352187318275 2024-05-08T17:00:21.858 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-08T17:00:21.859 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=352187318275 2024-05-08T17:00:21.859 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 352187318275' 2024-05-08T17:00:21.859 INFO:tasks.workunit.client.0.smithi172.stderr:waiting osd.3 seq 352187318275 2024-05-08T17:00:21.860 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-05-08T17:00:22.253 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 352187318275 -lt 352187318275 2024-05-08T17:00:22.253 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-05-08T17:00:22.253 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-05-08T17:00:22.254 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-05-08T17:00:22.794 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-05-08T17:00:22.794 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-05-08T17:00:22.795 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-05-08T17:00:22.795 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-05-08T17:00:22.795 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-05-08T17:00:22.795 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-05-08T17:00:22.796 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-05-08T17:00:22.796 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-05-08T17:00:23.184 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-05-08T17:00:23.184 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-05-08T17:00:23.185 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-05-08T17:00:23.185 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-05-08T17:00:23.705 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-05-08T17:00:23.705 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-05-08T17:00:23.706 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-05-08T17:00:23.706 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:209: set_size: dd if=/dev/urandom bs=256 count=1 2024-05-08T17:00:23.707 INFO:tasks.workunit.client.0.smithi172.stderr:1+0 records in 2024-05-08T17:00:23.707 INFO:tasks.workunit.client.0.smithi172.stderr:1+0 records out 2024-05-08T17:00:23.707 INFO:tasks.workunit.client.0.smithi172.stderr:256 bytes copied, 4.4582e-05 s, 5.7 MB/s 2024-05-08T17:00:23.707 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:216: set_size: objectstore_tool td/test-erasure-eio 3 obj-size-81110-0-256 set-bytes td/test-erasure-eio/CORRUPT 2024-05-08T17:00:23.708 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/test-erasure-eio 2024-05-08T17:00:23.708 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-05-08T17:00:23.708 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=3 2024-05-08T17:00:23.708 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-05-08T17:00:23.708 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/test-erasure-eio 3 obj-size-81110-0-256 set-bytes td/test-erasure-eio/CORRUPT 2024-05-08T17:00:23.708 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/test-erasure-eio 2024-05-08T17:00:23.708 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-05-08T17:00:23.708 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=3 2024-05-08T17:00:23.708 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-05-08T17:00:23.708 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/test-erasure-eio TERM osd.3 2024-05-08T17:00:23.709 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-08T17:00:23.709 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-08T17:00:23.709 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-08T17:00:23.709 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-08T17:00:23.709 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-08T17:00:24.019 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-08T17:00:24.019 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/test-erasure-eio 3 obj-size-81110-0-256 set-bytes td/test-erasure-eio/CORRUPT 2024-05-08T17:00:24.019 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/test-erasure-eio 2024-05-08T17:00:24.019 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-05-08T17:00:24.019 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=3 2024-05-08T17:00:24.020 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-05-08T17:00:24.020 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/test-erasure-eio/3 2024-05-08T17:00:24.020 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/test-erasure-eio/3 obj-size-81110-0-256 set-bytes td/test-erasure-eio/CORRUPT 2024-05-08T17:00:26.539 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/test-erasure-eio 3 2024-05-08T17:00:26.539 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/test-erasure-eio 2024-05-08T17:00:26.539 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-05-08T17:00:26.540 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2024-05-08T17:00:26.540 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-05-08T17:00:26.540 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/test-erasure-eio/3 2024-05-08T17:00:26.540 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=4457b0f1-028b-482b-8ee3-7ee05861e6cc --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-05-08T17:00:26.540 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-08T17:00:26.540 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-05-08T17:00:26.540 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-08T17:00:26.540 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/3' 2024-05-08T17:00:26.540 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/3/journal' 2024-05-08T17:00:26.540 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-05-08T17:00:26.540 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-05-08T17:00:26.540 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-05-08T17:00:26.541 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-05-08T17:00:26.541 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-08T17:00:26.541 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-08T17:00:26.541 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-08T17:00:26.541 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T17:00:26.542 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81110 2024-05-08T17:00:26.543 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.81110/$cluster-$name.asok' 2024-05-08T17:00:26.543 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.81110/$cluster-$name.asok' 2024-05-08T17:00:26.543 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-05-08T17:00:26.543 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-05-08T17:00:26.543 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-05-08T17:00:26.543 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-08T17:00:26.543 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-08T17:00:26.544 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-08T17:00:26.544 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-08T17:00:26.544 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-05-08T17:00:26.544 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-05-08T17:00:26.544 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/test-erasure-eio/3 2024-05-08T17:00:26.544 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2024-05-08T17:00:26.545 INFO:tasks.workunit.client.0.smithi172.stderr:start osd.3 2024-05-08T17:00:26.545 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=4457b0f1-028b-482b-8ee3-7ee05861e6cc --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81110/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-05-08T17:00:26.546 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/test-erasure-eio/3/whoami 2024-05-08T17:00:26.547 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2024-05-08T17:00:26.547 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-05-08T17:00:26.547 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-05-08T17:00:26.548 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-05-08T17:00:26.579 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:00:26.579+0000 7f6113b827c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T17:00:26.582 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:00:26.583+0000 7f6113b827c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T17:00:26.584 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:00:26.587+0000 7f6113b827c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T17:00:27.020 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2024-05-08T17:00:27.020 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-08T17:00:27.020 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-05-08T17:00:27.020 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-08T17:00:27.020 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-08T17:00:27.021 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:00:27.021 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-08T17:00:27.021 INFO:tasks.workunit.client.0.smithi172.stderr:0 2024-05-08T17:00:27.021 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:00:27.021 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T17:00:27.154 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:00:27.155+0000 7f6113b827c0 -1 Falling back to public interface 2024-05-08T17:00:27.460 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:00:28.462 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:00:28.463 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:00:28.463 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-08T17:00:28.463 INFO:tasks.workunit.client.0.smithi172.stderr:1 2024-05-08T17:00:28.463 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:00:28.463 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T17:00:28.861 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:00:29.862 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:00:29.863 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:00:29.863 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-08T17:00:29.863 INFO:tasks.workunit.client.0.smithi172.stderr:2 2024-05-08T17:00:29.863 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:00:29.863 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T17:00:29.865 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:00:29.867+0000 7f6113b827c0 -1 osd.3 84 log_to_monitors true 2024-05-08T17:00:30.315 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:00:31.317 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:00:31.317 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:00:31.317 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-05-08T17:00:31.317 INFO:tasks.workunit.client.0.smithi172.stderr:3 2024-05-08T17:00:31.317 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:00:31.317 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T17:00:31.913 INFO:tasks.workunit.client.0.smithi172.stderr:osd.3 up in weight 1 up_from 89 up_thru 89 down_at 85 last_clean_interval [82,84) [v2:127.0.0.1:6826/1948395538,v1:127.0.0.1:6827/1948395538] [v2:127.0.0.1:6828/1948395538,v1:127.0.0.1:6829/1948395538] exists,up 724678d7-ca35-4cdd-a1aa-910a2045b6f4 2024-05-08T17:00:31.913 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-08T17:00:31.913 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-08T17:00:31.913 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-08T17:00:31.913 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-05-08T17:00:31.913 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-05-08T17:00:31.913 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-05-08T17:00:31.913 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-05-08T17:00:31.913 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-05-08T17:00:31.914 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-05-08T17:00:31.914 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-05-08T17:00:31.914 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-05-08T17:00:31.914 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-05-08T17:00:31.914 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-05-08T17:00:32.033 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-05-08T17:00:32.033 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-05-08T17:00:32.033 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-05-08T17:00:32.034 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-05-08T17:00:32.034 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-05-08T17:00:32.034 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-05-08T17:00:32.457 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-05-08T17:00:32.457 INFO:tasks.workunit.client.0.smithi172.stderr:1 2024-05-08T17:00:32.457 INFO:tasks.workunit.client.0.smithi172.stderr:2 2024-05-08T17:00:32.457 INFO:tasks.workunit.client.0.smithi172.stderr:3' 2024-05-08T17:00:32.458 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-05-08T17:00:32.458 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T17:00:32.458 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-05-08T17:00:32.625 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=25769803839 2024-05-08T17:00:32.625 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 25769803839 2024-05-08T17:00:32.625 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803839' 2024-05-08T17:00:32.625 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T17:00:32.625 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-05-08T17:00:32.773 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=309237645325 2024-05-08T17:00:32.774 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 309237645325 2024-05-08T17:00:32.774 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803839 1-309237645325' 2024-05-08T17:00:32.774 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T17:00:32.774 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-05-08T17:00:32.921 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=85899345967 2024-05-08T17:00:32.922 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 85899345967 2024-05-08T17:00:32.922 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803839 1-309237645325 2-85899345967' 2024-05-08T17:00:32.922 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T17:00:32.922 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-05-08T17:00:33.072 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=382252089347 2024-05-08T17:00:33.072 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 382252089347 2024-05-08T17:00:33.072 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803839 1-309237645325 2-85899345967 3-382252089347' 2024-05-08T17:00:33.072 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-08T17:00:33.073 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-25769803839 2024-05-08T17:00:33.073 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-08T17:00:33.073 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-05-08T17:00:33.074 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-25769803839 2024-05-08T17:00:33.074 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-08T17:00:33.075 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=25769803839 2024-05-08T17:00:33.075 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 25769803839' 2024-05-08T17:00:33.075 INFO:tasks.workunit.client.0.smithi172.stderr:waiting osd.0 seq 25769803839 2024-05-08T17:00:33.076 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-05-08T17:00:33.461 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 25769803839 -lt 25769803839 2024-05-08T17:00:33.461 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-08T17:00:33.462 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-309237645325 2024-05-08T17:00:33.462 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-08T17:00:33.463 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-05-08T17:00:33.464 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-309237645325 2024-05-08T17:00:33.464 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-08T17:00:33.466 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=309237645325 2024-05-08T17:00:33.466 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 309237645325' 2024-05-08T17:00:33.466 INFO:tasks.workunit.client.0.smithi172.stderr:waiting osd.1 seq 309237645325 2024-05-08T17:00:33.466 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-05-08T17:00:33.884 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 309237645324 -lt 309237645325 2024-05-08T17:00:33.884 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-05-08T17:00:34.885 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-05-08T17:00:34.886 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-05-08T17:00:35.304 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 309237645325 -lt 309237645325 2024-05-08T17:00:35.304 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-08T17:00:35.305 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-85899345967 2024-05-08T17:00:35.305 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-08T17:00:35.307 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-05-08T17:00:35.308 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-85899345967 2024-05-08T17:00:35.308 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-08T17:00:35.309 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=85899345967 2024-05-08T17:00:35.310 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 85899345967' 2024-05-08T17:00:35.310 INFO:tasks.workunit.client.0.smithi172.stderr:waiting osd.2 seq 85899345967 2024-05-08T17:00:35.310 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-05-08T17:00:35.762 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 85899345968 -lt 85899345967 2024-05-08T17:00:35.762 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-08T17:00:35.763 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-382252089347 2024-05-08T17:00:35.763 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-08T17:00:35.764 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-05-08T17:00:35.765 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-382252089347 2024-05-08T17:00:35.765 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-08T17:00:35.767 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=382252089347 2024-05-08T17:00:35.767 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 382252089347' 2024-05-08T17:00:35.767 INFO:tasks.workunit.client.0.smithi172.stderr:waiting osd.3 seq 382252089347 2024-05-08T17:00:35.767 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-05-08T17:00:36.249 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 382252089347 -lt 382252089347 2024-05-08T17:00:36.249 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-05-08T17:00:36.250 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-05-08T17:00:36.250 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-05-08T17:00:36.808 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-05-08T17:00:36.808 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-05-08T17:00:36.809 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-05-08T17:00:36.809 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-05-08T17:00:36.809 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-05-08T17:00:36.809 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-05-08T17:00:36.809 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-05-08T17:00:36.809 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-05-08T17:00:37.275 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-05-08T17:00:37.275 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-05-08T17:00:37.276 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-05-08T17:00:37.276 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-05-08T17:00:37.813 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-05-08T17:00:37.813 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-05-08T17:00:37.813 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-05-08T17:00:37.813 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:217: set_size: rm -f td/test-erasure-eio/CORRUPT 2024-05-08T17:00:37.814 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:218: set_size: ceph osd unset noout 2024-05-08T17:00:38.506 INFO:tasks.workunit.client.0.smithi172.stderr:noout is unset 2024-05-08T17:00:38.528 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:238: rados_get_data_bad_size: rados_get td/test-erasure-eio pool-jerasure obj-size-81110-0-256 2024-05-08T17:00:38.529 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2024-05-08T17:00:38.529 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2024-05-08T17:00:38.529 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=obj-size-81110-0-256 2024-05-08T17:00:38.529 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=ok 2024-05-08T17:00:38.529 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' ok = fail ']' 2024-05-08T17:00:38.529 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:120: rados_get: rados --pool pool-jerasure get obj-size-81110-0-256 td/test-erasure-eio/COPY 2024-05-08T17:00:38.742 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:121: rados_get: diff td/test-erasure-eio/ORIGINAL td/test-erasure-eio/COPY 2024-05-08T17:00:38.743 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:122: rados_get: rm td/test-erasure-eio/COPY 2024-05-08T17:00:38.749 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:241: rados_get_data_bad_size: expr 0 + 1 2024-05-08T17:00:38.752 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:241: rados_get_data_bad_size: shard_id=1 2024-05-08T17:00:38.752 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:242: rados_get_data_bad_size: set_size obj-size-81110-0-256 td/test-erasure-eio 1 256 add 2024-05-08T17:00:38.752 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:192: set_size: local objname=obj-size-81110-0-256 2024-05-08T17:00:38.752 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:193: set_size: shift 2024-05-08T17:00:38.752 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:194: set_size: local dir=td/test-erasure-eio 2024-05-08T17:00:38.752 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:195: set_size: shift 2024-05-08T17:00:38.752 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:196: set_size: local shard_id=1 2024-05-08T17:00:38.752 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:197: set_size: shift 2024-05-08T17:00:38.752 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:198: set_size: local bytes=256 2024-05-08T17:00:38.752 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:199: set_size: shift 2024-05-08T17:00:38.752 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:200: set_size: local mode=add 2024-05-08T17:00:38.752 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:202: set_size: local poolname=pool-jerasure 2024-05-08T17:00:38.753 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: get_osds pool-jerasure obj-size-81110-0-256 2024-05-08T17:00:38.753 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2024-05-08T17:00:38.753 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-size-81110-0-256 2024-05-08T17:00:38.753 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure obj-size-81110-0-256 2024-05-08T17:00:38.753 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-05-08T17:00:39.196 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-05-08T17:00:39.197 INFO:tasks.workunit.client.0.smithi172.stderr:1 2024-05-08T17:00:39.197 INFO:tasks.workunit.client.0.smithi172.stderr:2' 2024-05-08T17:00:39.197 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 2 2024-05-08T17:00:39.197 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: initial_osds=('3' '1' '2') 2024-05-08T17:00:39.197 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: local -a initial_osds 2024-05-08T17:00:39.197 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:204: set_size: local osd_id=1 2024-05-08T17:00:39.197 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:205: set_size: ceph osd set noout 2024-05-08T17:00:39.870 INFO:tasks.workunit.client.0.smithi172.stderr:noout is set 2024-05-08T17:00:39.893 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:206: set_size: '[' add = add ']' 2024-05-08T17:00:39.893 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:208: set_size: objectstore_tool td/test-erasure-eio 1 obj-size-81110-0-256 get-bytes td/test-erasure-eio/CORRUPT 2024-05-08T17:00:39.893 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/test-erasure-eio 2024-05-08T17:00:39.893 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-05-08T17:00:39.893 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-05-08T17:00:39.893 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-05-08T17:00:39.893 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/test-erasure-eio 1 obj-size-81110-0-256 get-bytes td/test-erasure-eio/CORRUPT 2024-05-08T17:00:39.893 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/test-erasure-eio 2024-05-08T17:00:39.893 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-05-08T17:00:39.893 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-05-08T17:00:39.894 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-05-08T17:00:39.894 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/test-erasure-eio TERM osd.1 2024-05-08T17:00:39.894 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-08T17:00:39.894 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-08T17:00:39.895 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-08T17:00:39.895 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-08T17:00:39.895 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-08T17:00:41.207 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-08T17:00:41.207 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/test-erasure-eio 1 obj-size-81110-0-256 get-bytes td/test-erasure-eio/CORRUPT 2024-05-08T17:00:41.207 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/test-erasure-eio 2024-05-08T17:00:41.207 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-05-08T17:00:41.207 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-05-08T17:00:41.207 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-05-08T17:00:41.207 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/test-erasure-eio/1 2024-05-08T17:00:41.207 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/test-erasure-eio/1 obj-size-81110-0-256 get-bytes td/test-erasure-eio/CORRUPT 2024-05-08T17:00:43.080 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/test-erasure-eio 1 2024-05-08T17:00:43.080 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/test-erasure-eio 2024-05-08T17:00:43.080 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-05-08T17:00:43.080 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-05-08T17:00:43.080 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-05-08T17:00:43.080 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/test-erasure-eio/1 2024-05-08T17:00:43.080 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=4457b0f1-028b-482b-8ee3-7ee05861e6cc --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-05-08T17:00:43.080 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-08T17:00:43.080 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-05-08T17:00:43.081 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-08T17:00:43.081 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/1' 2024-05-08T17:00:43.081 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/1/journal' 2024-05-08T17:00:43.081 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-05-08T17:00:43.081 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-05-08T17:00:43.081 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-05-08T17:00:43.081 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-05-08T17:00:43.081 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-08T17:00:43.081 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-08T17:00:43.083 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-08T17:00:43.083 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T17:00:43.083 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81110 2024-05-08T17:00:43.083 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.81110/$cluster-$name.asok' 2024-05-08T17:00:43.083 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.81110/$cluster-$name.asok' 2024-05-08T17:00:43.083 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-05-08T17:00:43.083 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-05-08T17:00:43.083 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-05-08T17:00:43.083 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-08T17:00:43.084 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-08T17:00:43.084 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-08T17:00:43.084 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-08T17:00:43.084 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-05-08T17:00:43.084 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-05-08T17:00:43.084 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/test-erasure-eio/1 2024-05-08T17:00:43.084 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-05-08T17:00:43.084 INFO:tasks.workunit.client.0.smithi172.stderr:start osd.1 2024-05-08T17:00:43.084 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=4457b0f1-028b-482b-8ee3-7ee05861e6cc --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81110/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-05-08T17:00:43.085 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/test-erasure-eio/1/whoami 2024-05-08T17:00:43.085 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-05-08T17:00:43.086 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-05-08T17:00:43.086 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-05-08T17:00:43.086 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-05-08T17:00:43.117 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:00:43.119+0000 7fe361b297c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T17:00:43.120 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:00:43.123+0000 7fe361b297c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T17:00:43.121 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:00:43.123+0000 7fe361b297c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T17:00:43.497 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-05-08T17:00:43.497 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-08T17:00:43.497 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-05-08T17:00:43.497 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-08T17:00:43.497 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-08T17:00:43.497 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:00:43.498 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-08T17:00:43.498 INFO:tasks.workunit.client.0.smithi172.stderr:0 2024-05-08T17:00:43.498 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:00:43.498 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T17:00:43.682 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:00:43.687+0000 7fe361b297c0 -1 Falling back to public interface 2024-05-08T17:00:43.946 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:00:44.948 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:00:44.948 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:00:44.948 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-08T17:00:44.949 INFO:tasks.workunit.client.0.smithi172.stderr:1 2024-05-08T17:00:44.949 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:00:44.949 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T17:00:45.394 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:00:46.348 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:00:46.351+0000 7fe361b297c0 -1 osd.1 93 log_to_monitors true 2024-05-08T17:00:46.396 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:00:46.396 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:00:46.396 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-08T17:00:46.396 INFO:tasks.workunit.client.0.smithi172.stderr:2 2024-05-08T17:00:46.396 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:00:46.397 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T17:00:46.839 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:00:47.841 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:00:47.841 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:00:47.841 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-05-08T17:00:47.842 INFO:tasks.workunit.client.0.smithi172.stderr:3 2024-05-08T17:00:47.842 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:00:47.842 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T17:00:48.679 INFO:tasks.workunit.client.0.smithi172.stderr:osd.1 up in weight 1 up_from 100 up_thru 100 down_at 96 last_clean_interval [72,93) [v2:127.0.0.1:6810/65713091,v1:127.0.0.1:6811/65713091] [v2:127.0.0.1:6812/65713091,v1:127.0.0.1:6813/65713091] exists,up 0f51dcbd-8561-4314-b628-17fe241f145f 2024-05-08T17:00:48.679 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-08T17:00:48.679 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-08T17:00:48.679 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-08T17:00:48.679 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-05-08T17:00:48.679 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-05-08T17:00:48.679 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-05-08T17:00:48.679 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-05-08T17:00:48.680 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-05-08T17:00:48.680 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-05-08T17:00:48.680 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-05-08T17:00:48.680 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-05-08T17:00:48.681 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-05-08T17:00:48.681 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-05-08T17:00:48.799 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-05-08T17:00:48.799 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-05-08T17:00:48.800 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-05-08T17:00:48.800 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-05-08T17:00:48.800 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-05-08T17:00:48.800 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-05-08T17:00:49.232 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-05-08T17:00:49.233 INFO:tasks.workunit.client.0.smithi172.stderr:1 2024-05-08T17:00:49.233 INFO:tasks.workunit.client.0.smithi172.stderr:2 2024-05-08T17:00:49.233 INFO:tasks.workunit.client.0.smithi172.stderr:3' 2024-05-08T17:00:49.233 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-05-08T17:00:49.233 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T17:00:49.233 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-05-08T17:00:49.446 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=25769803844 2024-05-08T17:00:49.446 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 25769803844 2024-05-08T17:00:49.446 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803844' 2024-05-08T17:00:49.446 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T17:00:49.447 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-05-08T17:00:49.693 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=429496729603 2024-05-08T17:00:49.693 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 429496729603 2024-05-08T17:00:49.694 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803844 1-429496729603' 2024-05-08T17:00:49.694 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T17:00:49.694 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-05-08T17:00:49.853 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=85899345973 2024-05-08T17:00:49.853 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 85899345973 2024-05-08T17:00:49.853 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803844 1-429496729603 2-85899345973' 2024-05-08T17:00:49.853 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T17:00:49.853 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-05-08T17:00:50.007 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=382252089352 2024-05-08T17:00:50.007 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 382252089352 2024-05-08T17:00:50.007 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803844 1-429496729603 2-85899345973 3-382252089352' 2024-05-08T17:00:50.007 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-08T17:00:50.008 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-25769803844 2024-05-08T17:00:50.008 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-08T17:00:50.009 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-05-08T17:00:50.009 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-25769803844 2024-05-08T17:00:50.009 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-08T17:00:50.010 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=25769803844 2024-05-08T17:00:50.010 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 25769803844' 2024-05-08T17:00:50.011 INFO:tasks.workunit.client.0.smithi172.stderr:waiting osd.0 seq 25769803844 2024-05-08T17:00:50.011 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-05-08T17:00:50.412 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 25769803842 -lt 25769803844 2024-05-08T17:00:50.412 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-05-08T17:00:51.413 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-05-08T17:00:51.414 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-05-08T17:00:51.817 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 25769803845 -lt 25769803844 2024-05-08T17:00:51.817 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-08T17:00:51.818 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-08T17:00:51.818 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-429496729603 2024-05-08T17:00:51.819 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-05-08T17:00:51.820 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-429496729603 2024-05-08T17:00:51.820 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-08T17:00:51.821 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=429496729603 2024-05-08T17:00:51.822 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 429496729603' 2024-05-08T17:00:51.822 INFO:tasks.workunit.client.0.smithi172.stderr:waiting osd.1 seq 429496729603 2024-05-08T17:00:51.822 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-05-08T17:00:52.231 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 429496729603 -lt 429496729603 2024-05-08T17:00:52.231 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-08T17:00:52.232 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-85899345973 2024-05-08T17:00:52.232 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-08T17:00:52.234 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-05-08T17:00:52.235 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-85899345973 2024-05-08T17:00:52.235 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-08T17:00:52.236 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=85899345973 2024-05-08T17:00:52.237 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 85899345973' 2024-05-08T17:00:52.237 INFO:tasks.workunit.client.0.smithi172.stderr:waiting osd.2 seq 85899345973 2024-05-08T17:00:52.237 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-05-08T17:00:52.655 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 85899345973 -lt 85899345973 2024-05-08T17:00:52.656 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-08T17:00:52.657 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-08T17:00:52.657 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-382252089352 2024-05-08T17:00:52.658 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-05-08T17:00:52.659 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-382252089352 2024-05-08T17:00:52.659 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-08T17:00:52.660 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=382252089352 2024-05-08T17:00:52.660 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 382252089352' 2024-05-08T17:00:52.660 INFO:tasks.workunit.client.0.smithi172.stderr:waiting osd.3 seq 382252089352 2024-05-08T17:00:52.661 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-05-08T17:00:53.073 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 382252089353 -lt 382252089352 2024-05-08T17:00:53.074 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-05-08T17:00:53.074 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-05-08T17:00:53.075 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-05-08T17:00:53.591 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-05-08T17:00:53.591 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-05-08T17:00:53.592 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-05-08T17:00:53.592 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-05-08T17:00:53.592 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-05-08T17:00:53.592 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-05-08T17:00:53.593 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-05-08T17:00:53.593 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-05-08T17:00:54.025 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-05-08T17:00:54.026 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-05-08T17:00:54.027 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-05-08T17:00:54.027 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-05-08T17:00:54.557 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-05-08T17:00:54.558 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-05-08T17:00:54.558 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-05-08T17:00:54.558 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:209: set_size: dd if=/dev/urandom bs=256 count=1 2024-05-08T17:00:54.559 INFO:tasks.workunit.client.0.smithi172.stderr:1+0 records in 2024-05-08T17:00:54.559 INFO:tasks.workunit.client.0.smithi172.stderr:1+0 records out 2024-05-08T17:00:54.559 INFO:tasks.workunit.client.0.smithi172.stderr:256 bytes copied, 8.0575e-05 s, 3.2 MB/s 2024-05-08T17:00:54.561 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:216: set_size: objectstore_tool td/test-erasure-eio 1 obj-size-81110-0-256 set-bytes td/test-erasure-eio/CORRUPT 2024-05-08T17:00:54.561 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/test-erasure-eio 2024-05-08T17:00:54.561 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-05-08T17:00:54.561 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-05-08T17:00:54.561 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-05-08T17:00:54.561 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/test-erasure-eio 1 obj-size-81110-0-256 set-bytes td/test-erasure-eio/CORRUPT 2024-05-08T17:00:54.561 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/test-erasure-eio 2024-05-08T17:00:54.561 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-05-08T17:00:54.561 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-05-08T17:00:54.562 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-05-08T17:00:54.562 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/test-erasure-eio TERM osd.1 2024-05-08T17:00:54.562 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-08T17:00:54.562 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-08T17:00:54.562 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-08T17:00:54.562 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-08T17:00:54.562 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-08T17:00:54.871 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-08T17:00:54.871 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/test-erasure-eio 1 obj-size-81110-0-256 set-bytes td/test-erasure-eio/CORRUPT 2024-05-08T17:00:54.871 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/test-erasure-eio 2024-05-08T17:00:54.871 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-05-08T17:00:54.871 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-05-08T17:00:54.872 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-05-08T17:00:54.872 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/test-erasure-eio/1 2024-05-08T17:00:54.872 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/test-erasure-eio/1 obj-size-81110-0-256 set-bytes td/test-erasure-eio/CORRUPT 2024-05-08T17:00:57.394 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/test-erasure-eio 1 2024-05-08T17:00:57.394 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/test-erasure-eio 2024-05-08T17:00:57.395 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-05-08T17:00:57.395 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-05-08T17:00:57.395 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-05-08T17:00:57.395 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/test-erasure-eio/1 2024-05-08T17:00:57.395 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=4457b0f1-028b-482b-8ee3-7ee05861e6cc --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-05-08T17:00:57.395 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-08T17:00:57.395 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-05-08T17:00:57.395 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-08T17:00:57.395 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/1' 2024-05-08T17:00:57.395 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/1/journal' 2024-05-08T17:00:57.395 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-05-08T17:00:57.395 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-05-08T17:00:57.395 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-05-08T17:00:57.395 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-05-08T17:00:57.396 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-08T17:00:57.396 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-08T17:00:57.397 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-08T17:00:57.397 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T17:00:57.397 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81110 2024-05-08T17:00:57.397 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.81110/$cluster-$name.asok' 2024-05-08T17:00:57.398 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.81110/$cluster-$name.asok' 2024-05-08T17:00:57.398 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-05-08T17:00:57.398 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-05-08T17:00:57.398 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-05-08T17:00:57.398 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-08T17:00:57.398 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-08T17:00:57.398 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-08T17:00:57.398 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-08T17:00:57.398 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-05-08T17:00:57.398 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-05-08T17:00:57.398 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/test-erasure-eio/1 2024-05-08T17:00:57.399 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-05-08T17:00:57.399 INFO:tasks.workunit.client.0.smithi172.stderr:start osd.1 2024-05-08T17:00:57.399 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=4457b0f1-028b-482b-8ee3-7ee05861e6cc --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81110/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-05-08T17:00:57.399 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/test-erasure-eio/1/whoami 2024-05-08T17:00:57.400 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-05-08T17:00:57.401 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-05-08T17:00:57.401 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-05-08T17:00:57.401 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-05-08T17:00:57.432 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:00:57.439+0000 7fcfdb3b37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T17:00:57.435 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:00:57.439+0000 7fcfdb3b37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T17:00:57.436 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:00:57.443+0000 7fcfdb3b37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T17:00:57.824 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-05-08T17:00:57.824 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-08T17:00:57.824 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-05-08T17:00:57.824 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-08T17:00:57.824 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-08T17:00:57.824 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:00:57.824 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-08T17:00:57.824 INFO:tasks.workunit.client.0.smithi172.stderr:0 2024-05-08T17:00:57.824 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T17:00:57.824 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:00:57.993 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:00:57.999+0000 7fcfdb3b37c0 -1 Falling back to public interface 2024-05-08T17:00:58.275 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:00:59.276 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:00:59.276 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:00:59.277 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-08T17:00:59.277 INFO:tasks.workunit.client.0.smithi172.stderr:1 2024-05-08T17:00:59.277 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:00:59.277 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T17:00:59.674 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:01:00.659 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:01:00.663+0000 7fcfdb3b37c0 -1 osd.1 102 log_to_monitors true 2024-05-08T17:01:00.675 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:01:00.675 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:01:00.676 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-08T17:01:00.676 INFO:tasks.workunit.client.0.smithi172.stderr:2 2024-05-08T17:01:00.676 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:01:00.676 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T17:01:01.152 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:01:02.154 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:01:02.154 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:01:02.154 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-05-08T17:01:02.154 INFO:tasks.workunit.client.0.smithi172.stderr:3 2024-05-08T17:01:02.154 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:01:02.154 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T17:01:02.636 INFO:tasks.workunit.client.0.smithi172.stderr:osd.1 up in weight 1 up_from 107 up_thru 107 down_at 103 last_clean_interval [100,102) [v2:127.0.0.1:6810/2164729695,v1:127.0.0.1:6811/2164729695] [v2:127.0.0.1:6812/2164729695,v1:127.0.0.1:6813/2164729695] exists,up 0f51dcbd-8561-4314-b628-17fe241f145f 2024-05-08T17:01:02.637 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-08T17:01:02.637 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-08T17:01:02.637 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-08T17:01:02.637 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-05-08T17:01:02.637 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-05-08T17:01:02.637 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-05-08T17:01:02.637 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-05-08T17:01:02.637 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-05-08T17:01:02.638 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-05-08T17:01:02.638 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-05-08T17:01:02.639 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-05-08T17:01:02.639 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-05-08T17:01:02.639 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-05-08T17:01:02.748 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-05-08T17:01:02.748 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-05-08T17:01:02.748 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-05-08T17:01:02.748 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-05-08T17:01:02.748 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-05-08T17:01:02.748 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-05-08T17:01:03.181 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-05-08T17:01:03.181 INFO:tasks.workunit.client.0.smithi172.stderr:1 2024-05-08T17:01:03.181 INFO:tasks.workunit.client.0.smithi172.stderr:2 2024-05-08T17:01:03.181 INFO:tasks.workunit.client.0.smithi172.stderr:3' 2024-05-08T17:01:03.181 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-05-08T17:01:03.181 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T17:01:03.181 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-05-08T17:01:03.333 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=25769803849 2024-05-08T17:01:03.333 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 25769803849 2024-05-08T17:01:03.333 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803849' 2024-05-08T17:01:03.334 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T17:01:03.334 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-05-08T17:01:03.487 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=459561500675 2024-05-08T17:01:03.489 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 459561500675 2024-05-08T17:01:03.489 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803849 1-459561500675' 2024-05-08T17:01:03.489 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T17:01:03.490 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-05-08T17:01:03.632 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=85899345978 2024-05-08T17:01:03.632 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 85899345978 2024-05-08T17:01:03.633 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803849 1-459561500675 2-85899345978' 2024-05-08T17:01:03.633 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T17:01:03.633 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-05-08T17:01:03.774 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=382252089357 2024-05-08T17:01:03.774 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 382252089357 2024-05-08T17:01:03.774 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803849 1-459561500675 2-85899345978 3-382252089357' 2024-05-08T17:01:03.774 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-08T17:01:03.775 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-25769803849 2024-05-08T17:01:03.775 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-08T17:01:03.775 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-05-08T17:01:03.776 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-25769803849 2024-05-08T17:01:03.777 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-08T17:01:03.777 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=25769803849 2024-05-08T17:01:03.777 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 25769803849' 2024-05-08T17:01:03.778 INFO:tasks.workunit.client.0.smithi172.stderr:waiting osd.0 seq 25769803849 2024-05-08T17:01:03.778 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-05-08T17:01:04.187 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 25769803847 -lt 25769803849 2024-05-08T17:01:04.187 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-05-08T17:01:05.189 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-05-08T17:01:05.190 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-05-08T17:01:05.589 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 25769803849 -lt 25769803849 2024-05-08T17:01:05.589 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-08T17:01:05.590 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-459561500675 2024-05-08T17:01:05.590 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-08T17:01:05.592 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-05-08T17:01:05.592 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-459561500675 2024-05-08T17:01:05.592 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-08T17:01:05.594 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=459561500675 2024-05-08T17:01:05.594 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 459561500675' 2024-05-08T17:01:05.594 INFO:tasks.workunit.client.0.smithi172.stderr:waiting osd.1 seq 459561500675 2024-05-08T17:01:05.594 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-05-08T17:01:06.019 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 459561500675 -lt 459561500675 2024-05-08T17:01:06.019 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-08T17:01:06.020 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-85899345978 2024-05-08T17:01:06.020 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-08T17:01:06.021 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-05-08T17:01:06.022 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-85899345978 2024-05-08T17:01:06.022 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-08T17:01:06.024 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=85899345978 2024-05-08T17:01:06.025 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 85899345978' 2024-05-08T17:01:06.025 INFO:tasks.workunit.client.0.smithi172.stderr:waiting osd.2 seq 85899345978 2024-05-08T17:01:06.025 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-05-08T17:01:06.453 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 85899345978 -lt 85899345978 2024-05-08T17:01:06.453 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-08T17:01:06.454 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-382252089357 2024-05-08T17:01:06.454 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-08T17:01:06.455 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-05-08T17:01:06.456 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-382252089357 2024-05-08T17:01:06.456 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-08T17:01:06.457 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=382252089357 2024-05-08T17:01:06.457 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 382252089357' 2024-05-08T17:01:06.457 INFO:tasks.workunit.client.0.smithi172.stderr:waiting osd.3 seq 382252089357 2024-05-08T17:01:06.457 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-05-08T17:01:06.914 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 382252089358 -lt 382252089357 2024-05-08T17:01:06.914 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-05-08T17:01:06.915 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-05-08T17:01:06.915 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-05-08T17:01:07.456 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-05-08T17:01:07.456 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-05-08T17:01:07.457 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-05-08T17:01:07.457 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-05-08T17:01:07.457 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-05-08T17:01:07.457 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-05-08T17:01:07.457 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-05-08T17:01:07.458 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-05-08T17:01:07.840 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-05-08T17:01:07.840 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-05-08T17:01:07.840 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-05-08T17:01:07.840 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-05-08T17:01:08.363 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-05-08T17:01:08.363 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-05-08T17:01:08.364 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-05-08T17:01:08.364 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:217: set_size: rm -f td/test-erasure-eio/CORRUPT 2024-05-08T17:01:08.364 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:218: set_size: ceph osd unset noout 2024-05-08T17:01:09.103 INFO:tasks.workunit.client.0.smithi172.stderr:noout is unset 2024-05-08T17:01:09.126 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:243: rados_get_data_bad_size: rados_get td/test-erasure-eio pool-jerasure obj-size-81110-0-256 fail 2024-05-08T17:01:09.126 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2024-05-08T17:01:09.126 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2024-05-08T17:01:09.126 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=obj-size-81110-0-256 2024-05-08T17:01:09.126 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=fail 2024-05-08T17:01:09.126 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' fail = fail ']' 2024-05-08T17:01:09.126 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:114: rados_get: rados --pool pool-jerasure get obj-size-81110-0-256 td/test-erasure-eio/COPY 2024-05-08T17:01:09.303 INFO:tasks.workunit.client.0.smithi172.stderr:error getting pool-jerasure/obj-size-81110-0-256: (5) Input/output error 2024-05-08T17:01:09.307 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:115: rados_get: return 2024-05-08T17:01:09.307 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:244: rados_get_data_bad_size: rm td/test-erasure-eio/ORIGINAL 2024-05-08T17:01:09.309 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:312: TEST_rados_get_bad_size_shard_0: delete_erasure_coded_pool pool-jerasure 2024-05-08T17:01:09.309 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:84: delete_erasure_coded_pool: local poolname=pool-jerasure 2024-05-08T17:01:09.309 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:85: delete_erasure_coded_pool: ceph osd pool delete pool-jerasure pool-jerasure --yes-i-really-really-mean-it 2024-05-08T17:01:09.811 INFO:tasks.workunit.client.0.smithi172.stderr:pool 'pool-jerasure' does not exist 2024-05-08T17:01:09.826 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:86: delete_erasure_coded_pool: ceph osd erasure-code-profile rm myprofile 2024-05-08T17:01:10.541 INFO:tasks.workunit.client.0.smithi172.stderr:erasure-code-profile myprofile does not exist 2024-05-08T17:01:10.558 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:42: run: teardown td/test-erasure-eio 2024-05-08T17:01:10.558 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-eio 2024-05-08T17:01:10.558 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-05-08T17:01:10.558 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-eio KILL 2024-05-08T17:01:10.559 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-08T17:01:10.559 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-08T17:01:10.559 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-08T17:01:10.559 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-08T17:01:10.559 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-08T17:01:10.870 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-08T17:01:10.870 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-05-08T17:01:10.871 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-05-08T17:01:10.872 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-05-08T17:01:10.873 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-05-08T17:01:10.873 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-05-08T17:01:10.874 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-05-08T17:01:10.875 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-08T17:01:10.876 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-05-08T17:01:10.876 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-05-08T17:01:10.876 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-08T17:01:10.877 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-05-08T17:01:10.878 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-05-08T17:01:10.879 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-eio 2024-05-08T17:01:10.914 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-05-08T17:01:10.914 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T17:01:10.914 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81110 2024-05-08T17:01:10.914 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.81110 2024-05-08T17:01:10.916 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-05-08T17:01:10.916 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-05-08T17:01:10.916 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:32: run: for func in $funcs 2024-05-08T17:01:10.916 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:33: run: setup td/test-erasure-eio 2024-05-08T17:01:10.916 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/test-erasure-eio 2024-05-08T17:01:10.916 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/test-erasure-eio 2024-05-08T17:01:10.916 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-eio 2024-05-08T17:01:10.916 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-05-08T17:01:10.917 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-eio KILL 2024-05-08T17:01:10.917 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-08T17:01:10.917 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-08T17:01:10.917 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-08T17:01:10.917 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-08T17:01:10.917 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-08T17:01:10.918 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-08T17:01:10.919 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-05-08T17:01:10.919 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-05-08T17:01:10.920 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-05-08T17:01:10.921 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-05-08T17:01:10.921 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-05-08T17:01:10.921 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-05-08T17:01:10.922 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-08T17:01:10.922 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-05-08T17:01:10.923 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-05-08T17:01:10.923 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-08T17:01:10.924 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-05-08T17:01:10.925 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-05-08T17:01:10.925 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-eio 2024-05-08T17:01:10.927 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-05-08T17:01:10.927 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T17:01:10.927 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81110 2024-05-08T17:01:10.927 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.81110 2024-05-08T17:01:10.927 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-05-08T17:01:10.928 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-05-08T17:01:10.928 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/test-erasure-eio 2024-05-08T17:01:10.929 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-05-08T17:01:10.930 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T17:01:10.930 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81110 2024-05-08T17:01:10.930 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.81110 2024-05-08T17:01:10.932 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-05-08T17:01:10.932 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-05-08T17:01:10.932 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-05-08T17:01:10.932 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/test-erasure-eio 1' TERM HUP INT 2024-05-08T17:01:10.932 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:34: run: run_mon td/test-erasure-eio a 2024-05-08T17:01:10.932 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/test-erasure-eio 2024-05-08T17:01:10.932 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-05-08T17:01:10.932 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-05-08T17:01:10.932 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-05-08T17:01:10.932 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/test-erasure-eio/a 2024-05-08T17:01:10.932 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-eio/a --run-dir=td/test-erasure-eio 2024-05-08T17:01:11.336 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-05-08T17:01:11.336 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-08T17:01:11.337 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-08T17:01:11.337 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-08T17:01:11.337 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T17:01:11.337 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81110 2024-05-08T17:01:11.338 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.81110/$cluster-$name.asok' 2024-05-08T17:01:11.338 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-eio/a '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.81110/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-eio/log --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-05-08T17:01:11.564 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-05-08T17:01:11.566 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-05-08T17:01:11.566 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-05-08T17:01:11.566 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-05-08T17:01:11.566 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-05-08T17:01:11.567 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-05-08T17:01:11.567 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-05-08T17:01:11.567 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-05-08T17:01:11.567 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-05-08T17:01:11.567 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-08T17:01:11.567 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T17:01:11.567 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81110 2024-05-08T17:01:11.568 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.81110/ceph-mon.a.asok 2024-05-08T17:01:11.568 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-05-08T17:01:11.568 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.81110/ceph-mon.a.asok config get fsid 2024-05-08T17:01:11.681 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-05-08T17:01:11.681 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-05-08T17:01:11.681 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-05-08T17:01:11.681 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-05-08T17:01:11.682 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-05-08T17:01:11.682 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-05-08T17:01:11.682 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-05-08T17:01:11.682 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-05-08T17:01:11.686 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-08T17:01:11.686 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T17:01:11.686 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81110 2024-05-08T17:01:11.686 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.81110/ceph-mon.a.asok 2024-05-08T17:01:11.687 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-05-08T17:01:11.687 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.81110/ceph-mon.a.asok config get mon_host 2024-05-08T17:01:11.785 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:35: run: run_mgr td/test-erasure-eio x 2024-05-08T17:01:11.785 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/test-erasure-eio 2024-05-08T17:01:11.785 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-05-08T17:01:11.785 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-05-08T17:01:11.785 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-05-08T17:01:11.786 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/test-erasure-eio/x 2024-05-08T17:01:11.786 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-05-08T17:01:12.707 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-05-08T17:01:12.707 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-08T17:01:12.707 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-08T17:01:12.707 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-08T17:01:12.707 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T17:01:12.707 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81110 2024-05-08T17:01:12.707 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.81110/$cluster-$name.asok' 2024-05-08T17:01:12.707 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-05-08T17:01:12.708 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-eio/x '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.81110/$cluster-$name.asok' --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-05-08T17:01:12.739 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:36: run: create_pool rbd 4 2024-05-08T17:01:12.739 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-05-08T17:01:13.059 INFO:tasks.workunit.client.0.smithi172.stderr:pool 'rbd' already exists 2024-05-08T17:01:13.076 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-05-08T17:01:14.080 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: get_asok_path mon.a 2024-05-08T17:01:14.080 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-05-08T17:01:14.080 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-05-08T17:01:14.080 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-08T17:01:14.080 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T17:01:14.081 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81110 2024-05-08T17:01:14.081 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.81110/ceph-mon.a.asok 2024-05-08T17:01:14.081 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: CEPH_ARGS= 2024-05-08T17:01:14.081 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: ceph --admin-daemon /tmp/ceph-asok.81110/ceph-mon.a.asok log flush 2024-05-08T17:01:14.188 INFO:tasks.workunit.client.0.smithi172.stdout:{} 2024-05-08T17:01:14.200 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:40: run: grep 'load: jerasure.*lrc' td/test-erasure-eio/mon.a.log 2024-05-08T17:01:14.203 INFO:tasks.workunit.client.0.smithi172.stdout:2024-05-08T17:01:11.387+0000 7f397d83ec80 0 load: jerasure load: lrc 2024-05-08T17:01:14.204 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:41: run: TEST_rados_get_bad_size_shard_1 td/test-erasure-eio 2024-05-08T17:01:14.204 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:316: TEST_rados_get_bad_size_shard_1: local dir=td/test-erasure-eio 2024-05-08T17:01:14.204 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:317: TEST_rados_get_bad_size_shard_1: setup_osds 4 2024-05-08T17:01:14.204 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:47: setup_osds: local count=4 2024-05-08T17:01:14.204 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:48: setup_osds: shift 2024-05-08T17:01:14.205 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: expr 4 - 1 2024-05-08T17:01:14.206 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: seq 0 3 2024-05-08T17:01:14.208 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-05-08T17:01:14.208 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 0 2024-05-08T17:01:14.209 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-05-08T17:01:14.209 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-05-08T17:01:14.209 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-05-08T17:01:14.209 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-05-08T17:01:14.209 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/0 2024-05-08T17:01:14.209 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=4457b0f1-028b-482b-8ee3-7ee05861e6cc --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-05-08T17:01:14.209 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-08T17:01:14.209 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-05-08T17:01:14.209 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-08T17:01:14.209 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/0' 2024-05-08T17:01:14.209 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/0/journal' 2024-05-08T17:01:14.209 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-05-08T17:01:14.209 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-05-08T17:01:14.209 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-05-08T17:01:14.209 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-05-08T17:01:14.210 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-08T17:01:14.210 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-08T17:01:14.210 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-08T17:01:14.210 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T17:01:14.210 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81110 2024-05-08T17:01:14.210 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.81110/$cluster-$name.asok' 2024-05-08T17:01:14.211 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.81110/$cluster-$name.asok' 2024-05-08T17:01:14.211 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-05-08T17:01:14.211 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-05-08T17:01:14.211 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-05-08T17:01:14.211 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-05-08T17:01:14.211 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-05-08T17:01:14.211 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-08T17:01:14.211 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-08T17:01:14.211 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-08T17:01:14.211 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-08T17:01:14.211 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-05-08T17:01:14.212 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-05-08T17:01:14.212 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/0 2024-05-08T17:01:14.212 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-05-08T17:01:14.212 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=3448315a-530f-465b-b4af-1e3c6d7d96eb 2024-05-08T17:01:14.212 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 3448315a-530f-465b-b4af-1e3c6d7d96eb' 2024-05-08T17:01:14.213 INFO:tasks.workunit.client.0.smithi172.stdout:add osd0 3448315a-530f-465b-b4af-1e3c6d7d96eb 2024-05-08T17:01:14.213 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-05-08T17:01:14.234 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDarztmnLdQDhAAWeOG12ST2P8kx2WOQFwpcA== 2024-05-08T17:01:14.234 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDarztmnLdQDhAAWeOG12ST2P8kx2WOQFwpcA=="}' 2024-05-08T17:01:14.234 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 3448315a-530f-465b-b4af-1e3c6d7d96eb -i td/test-erasure-eio/0/new.json 2024-05-08T17:01:14.518 INFO:tasks.workunit.client.0.smithi172.stdout:0 2024-05-08T17:01:14.533 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/0/new.json 2024-05-08T17:01:14.534 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=4457b0f1-028b-482b-8ee3-7ee05861e6cc --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81110/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDarztmnLdQDhAAWeOG12ST2P8kx2WOQFwpcA== --osd-uuid 3448315a-530f-465b-b4af-1e3c6d7d96eb 2024-05-08T17:01:14.568 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:01:14.574+0000 7f4fca4d77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T17:01:14.586 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:01:14.590+0000 7f4fca4d77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T17:01:14.587 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:01:14.590+0000 7f4fca4d77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T17:01:14.587 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:01:14.590+0000 7f4fca4d77c0 -1 bluestore(td/test-erasure-eio/0/block) _read_bdev_label failed to open td/test-erasure-eio/0/block: (2) No such file or directory 2024-05-08T17:01:14.588 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:01:14.590+0000 7f4fca4d77c0 -1 bluestore(td/test-erasure-eio/0/block) _read_bdev_label failed to open td/test-erasure-eio/0/block: (2) No such file or directory 2024-05-08T17:01:14.588 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:01:14.590+0000 7f4fca4d77c0 -1 bluestore(td/test-erasure-eio/0/block) _read_bdev_label failed to open td/test-erasure-eio/0/block: (2) No such file or directory 2024-05-08T17:01:14.626 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:01:14.630+0000 7f4fca4d77c0 -1 bluestore(td/test-erasure-eio/0) _read_fsid unparsable uuid 2024-05-08T17:01:21.633 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/0/keyring 2024-05-08T17:01:21.633 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-05-08T17:01:21.635 INFO:tasks.workunit.client.0.smithi172.stdout:adding osd0 key to auth repository 2024-05-08T17:01:21.635 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-05-08T17:01:21.635 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-05-08T17:01:22.188 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-05-08T17:01:22.188 INFO:tasks.workunit.client.0.smithi172.stdout:start osd.0 2024-05-08T17:01:22.189 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=4457b0f1-028b-482b-8ee3-7ee05861e6cc --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81110/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-05-08T17:01:22.189 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-05-08T17:01:22.189 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-05-08T17:01:22.189 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-05-08T17:01:22.214 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:01:22.218+0000 7f1240bc27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T17:01:22.216 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:01:22.222+0000 7f1240bc27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T17:01:22.218 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:01:22.222+0000 7f1240bc27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T17:01:22.578 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-05-08T17:01:22.578 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-08T17:01:22.578 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-05-08T17:01:22.578 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-08T17:01:22.578 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-08T17:01:22.578 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:01:22.578 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-08T17:01:22.579 INFO:tasks.workunit.client.0.smithi172.stdout:0 2024-05-08T17:01:22.579 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T17:01:22.579 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:01:22.770 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:01:22.774+0000 7f1240bc27c0 -1 Falling back to public interface 2024-05-08T17:01:23.021 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:01:24.023 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:01:24.023 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:01:24.023 INFO:tasks.workunit.client.0.smithi172.stdout:1 2024-05-08T17:01:24.024 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-08T17:01:24.024 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:01:24.024 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T17:01:24.435 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:01:25.431 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:01:25.434+0000 7f1240bc27c0 -1 osd.0 0 log_to_monitors true 2024-05-08T17:01:25.436 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:01:25.436 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:01:25.437 INFO:tasks.workunit.client.0.smithi172.stdout:2 2024-05-08T17:01:25.437 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-08T17:01:25.437 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:01:25.437 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T17:01:25.974 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:01:26.976 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:01:26.976 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:01:26.977 INFO:tasks.workunit.client.0.smithi172.stdout:3 2024-05-08T17:01:26.977 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-05-08T17:01:26.977 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:01:26.977 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T17:01:27.474 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:01:28.476 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:01:28.476 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:01:28.476 INFO:tasks.workunit.client.0.smithi172.stdout:4 2024-05-08T17:01:28.477 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-05-08T17:01:28.477 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:01:28.477 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T17:01:28.906 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:01:29.906 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:01:29.906 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:01:29.907 INFO:tasks.workunit.client.0.smithi172.stdout:5 2024-05-08T17:01:29.907 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-05-08T17:01:29.907 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:01:29.907 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T17:01:30.460 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:01:31.463 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:01:31.463 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:01:31.463 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-05-08T17:01:31.463 INFO:tasks.workunit.client.0.smithi172.stdout:6 2024-05-08T17:01:31.463 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:01:31.464 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T17:01:31.881 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:01:32.882 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:01:32.884 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:01:32.884 INFO:tasks.workunit.client.0.smithi172.stdout:7 2024-05-08T17:01:32.885 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-05-08T17:01:32.885 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:01:32.885 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T17:01:33.461 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:01:34.463 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:01:34.463 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:01:34.464 INFO:tasks.workunit.client.0.smithi172.stdout:8 2024-05-08T17:01:34.464 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-05-08T17:01:34.464 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:01:34.464 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T17:01:34.866 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:01:35.868 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:01:35.868 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:01:35.868 INFO:tasks.workunit.client.0.smithi172.stdout:9 2024-05-08T17:01:35.869 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-05-08T17:01:35.869 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:01:35.869 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T17:01:36.373 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:01:37.375 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:01:37.376 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:01:37.376 INFO:tasks.workunit.client.0.smithi172.stdout:10 2024-05-08T17:01:37.376 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-05-08T17:01:37.377 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:01:37.377 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T17:01:37.903 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:01:38.904 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:01:38.905 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:01:38.905 INFO:tasks.workunit.client.0.smithi172.stdout:11 2024-05-08T17:01:38.905 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-05-08T17:01:38.905 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:01:38.906 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T17:01:39.897 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:01:40.899 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:01:40.899 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:01:40.899 INFO:tasks.workunit.client.0.smithi172.stdout:12 2024-05-08T17:01:40.900 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-05-08T17:01:40.900 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:01:40.900 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T17:01:41.711 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:01:42.713 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:01:42.713 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:01:42.713 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-05-08T17:01:42.714 INFO:tasks.workunit.client.0.smithi172.stdout:13 2024-05-08T17:01:42.714 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:01:42.714 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T17:01:43.494 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:01:44.496 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:01:44.496 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:01:44.496 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-05-08T17:01:44.496 INFO:tasks.workunit.client.0.smithi172.stdout:14 2024-05-08T17:01:44.497 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:01:44.497 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T17:01:45.230 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:01:46.232 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:01:46.232 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:01:46.232 INFO:tasks.workunit.client.0.smithi172.stdout:15 2024-05-08T17:01:46.233 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-05-08T17:01:46.233 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:01:46.233 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T17:01:47.329 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:01:48.331 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:01:48.331 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:01:48.331 INFO:tasks.workunit.client.0.smithi172.stdout:16 2024-05-08T17:01:48.332 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-05-08T17:01:48.332 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:01:48.332 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T17:01:49.231 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:01:50.233 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:01:50.233 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:01:50.233 INFO:tasks.workunit.client.0.smithi172.stdout:17 2024-05-08T17:01:50.233 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-05-08T17:01:50.234 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:01:50.234 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T17:01:50.917 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:01:51.919 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:01:51.919 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:01:51.919 INFO:tasks.workunit.client.0.smithi172.stdout:18 2024-05-08T17:01:51.920 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-05-08T17:01:51.920 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:01:51.920 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T17:01:52.412 INFO:tasks.workunit.client.0.smithi172.stdout:osd.0 up in weight 1 up_from 6 up_thru 6 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3007160394,v1:127.0.0.1:6803/3007160394] [v2:127.0.0.1:6804/3007160394,v1:127.0.0.1:6805/3007160394] exists,up 3448315a-530f-465b-b4af-1e3c6d7d96eb 2024-05-08T17:01:52.416 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-08T17:01:52.416 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-08T17:01:52.416 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-08T17:01:52.416 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-05-08T17:01:52.416 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 1 2024-05-08T17:01:52.416 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-05-08T17:01:52.416 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-05-08T17:01:52.416 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-05-08T17:01:52.416 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-05-08T17:01:52.417 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/1 2024-05-08T17:01:52.417 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=4457b0f1-028b-482b-8ee3-7ee05861e6cc --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-05-08T17:01:52.417 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-08T17:01:52.417 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-05-08T17:01:52.417 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-08T17:01:52.417 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/1' 2024-05-08T17:01:52.417 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/1/journal' 2024-05-08T17:01:52.417 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-05-08T17:01:52.417 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-05-08T17:01:52.417 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-05-08T17:01:52.417 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-05-08T17:01:52.417 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-08T17:01:52.417 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-08T17:01:52.417 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-08T17:01:52.418 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T17:01:52.418 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81110 2024-05-08T17:01:52.418 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.81110/$cluster-$name.asok' 2024-05-08T17:01:52.418 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.81110/$cluster-$name.asok' 2024-05-08T17:01:52.418 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-05-08T17:01:52.418 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-05-08T17:01:52.418 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-05-08T17:01:52.418 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-05-08T17:01:52.418 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-05-08T17:01:52.418 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-08T17:01:52.418 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-08T17:01:52.418 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-08T17:01:52.418 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-08T17:01:52.418 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-05-08T17:01:52.419 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-05-08T17:01:52.419 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/1 2024-05-08T17:01:52.419 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-05-08T17:01:52.419 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=a0a953bf-e981-4e54-97d5-d7c2101753a9 2024-05-08T17:01:52.419 INFO:tasks.workunit.client.0.smithi172.stdout:add osd1 a0a953bf-e981-4e54-97d5-d7c2101753a9 2024-05-08T17:01:52.420 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 a0a953bf-e981-4e54-97d5-d7c2101753a9' 2024-05-08T17:01:52.420 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-05-08T17:01:52.434 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAAsDtmlP1FGhAAvn3pW+scnnG7/ykxG9R03A== 2024-05-08T17:01:52.434 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAAsDtmlP1FGhAAvn3pW+scnnG7/ykxG9R03A=="}' 2024-05-08T17:01:52.434 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new a0a953bf-e981-4e54-97d5-d7c2101753a9 -i td/test-erasure-eio/1/new.json 2024-05-08T17:01:52.911 INFO:tasks.workunit.client.0.smithi172.stdout:1 2024-05-08T17:01:52.926 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/1/new.json 2024-05-08T17:01:52.928 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=4457b0f1-028b-482b-8ee3-7ee05861e6cc --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81110/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAAsDtmlP1FGhAAvn3pW+scnnG7/ykxG9R03A== --osd-uuid a0a953bf-e981-4e54-97d5-d7c2101753a9 2024-05-08T17:01:52.957 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:01:52.962+0000 7f93345117c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T17:01:52.968 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:01:52.974+0000 7f93345117c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T17:01:52.969 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:01:52.974+0000 7f93345117c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T17:01:52.970 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:01:52.974+0000 7f93345117c0 -1 bluestore(td/test-erasure-eio/1/block) _read_bdev_label failed to open td/test-erasure-eio/1/block: (2) No such file or directory 2024-05-08T17:01:52.970 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:01:52.974+0000 7f93345117c0 -1 bluestore(td/test-erasure-eio/1/block) _read_bdev_label failed to open td/test-erasure-eio/1/block: (2) No such file or directory 2024-05-08T17:01:52.970 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:01:52.974+0000 7f93345117c0 -1 bluestore(td/test-erasure-eio/1/block) _read_bdev_label failed to open td/test-erasure-eio/1/block: (2) No such file or directory 2024-05-08T17:01:53.008 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:01:53.014+0000 7f93345117c0 -1 bluestore(td/test-erasure-eio/1) _read_fsid unparsable uuid 2024-05-08T17:01:59.745 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/1/keyring 2024-05-08T17:01:59.745 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-05-08T17:01:59.746 INFO:tasks.workunit.client.0.smithi172.stdout:adding osd1 key to auth repository 2024-05-08T17:01:59.747 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-05-08T17:01:59.747 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-05-08T17:02:00.325 INFO:tasks.workunit.client.0.smithi172.stdout:start osd.1 2024-05-08T17:02:00.326 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-05-08T17:02:00.326 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-05-08T17:02:00.327 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=4457b0f1-028b-482b-8ee3-7ee05861e6cc --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81110/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-05-08T17:02:00.327 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-05-08T17:02:00.327 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-05-08T17:02:00.358 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:02:00.362+0000 7f1a5b3247c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T17:02:00.360 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:02:00.366+0000 7f1a5b3247c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T17:02:00.361 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:02:00.366+0000 7f1a5b3247c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T17:02:00.751 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-05-08T17:02:00.751 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-08T17:02:00.751 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-05-08T17:02:00.752 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-08T17:02:00.752 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-08T17:02:00.752 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:02:00.752 INFO:tasks.workunit.client.0.smithi172.stdout:0 2024-05-08T17:02:00.752 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-08T17:02:00.752 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:02:00.753 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T17:02:00.917 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:02:00.922+0000 7f1a5b3247c0 -1 Falling back to public interface 2024-05-08T17:02:01.189 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:02:02.191 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:02:02.191 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:02:02.192 INFO:tasks.workunit.client.0.smithi172.stdout:1 2024-05-08T17:02:02.192 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-08T17:02:02.192 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:02:02.192 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T17:02:02.607 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:02:03.609 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:02:03.609 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:02:03.609 INFO:tasks.workunit.client.0.smithi172.stdout:2 2024-05-08T17:02:03.610 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-08T17:02:03.610 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:02:03.610 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T17:02:03.648 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:02:03.654+0000 7f1a5b3247c0 -1 osd.1 0 log_to_monitors true 2024-05-08T17:02:04.205 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:02:05.206 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:02:05.207 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:02:05.207 INFO:tasks.workunit.client.0.smithi172.stdout:3 2024-05-08T17:02:05.207 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-05-08T17:02:05.207 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:02:05.208 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T17:02:05.848 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:02:06.851 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:02:06.851 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:02:06.851 INFO:tasks.workunit.client.0.smithi172.stdout:4 2024-05-08T17:02:06.852 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-05-08T17:02:06.852 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:02:06.852 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T17:02:07.267 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:02:08.269 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:02:08.269 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:02:08.269 INFO:tasks.workunit.client.0.smithi172.stdout:5 2024-05-08T17:02:08.270 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-05-08T17:02:08.270 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:02:08.270 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T17:02:08.774 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:02:09.776 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:02:09.776 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:02:09.777 INFO:tasks.workunit.client.0.smithi172.stdout:6 2024-05-08T17:02:09.777 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-05-08T17:02:09.777 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:02:09.777 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T17:02:10.318 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:02:11.320 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:02:11.320 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:02:11.320 INFO:tasks.workunit.client.0.smithi172.stdout:7 2024-05-08T17:02:11.321 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-05-08T17:02:11.321 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:02:11.321 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T17:02:12.117 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:02:13.118 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:02:13.118 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:02:13.118 INFO:tasks.workunit.client.0.smithi172.stdout:8 2024-05-08T17:02:13.119 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-05-08T17:02:13.119 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:02:13.119 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T17:02:13.733 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:02:14.735 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:02:14.735 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:02:14.746 INFO:tasks.workunit.client.0.smithi172.stdout:9 2024-05-08T17:02:14.747 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-05-08T17:02:14.747 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:02:14.747 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T17:02:15.185 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:02:16.186 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:02:16.186 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:02:16.187 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-05-08T17:02:16.187 INFO:tasks.workunit.client.0.smithi172.stdout:10 2024-05-08T17:02:16.188 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:02:16.188 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T17:02:16.775 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:02:17.777 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:02:17.777 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:02:17.777 INFO:tasks.workunit.client.0.smithi172.stdout:11 2024-05-08T17:02:17.778 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-05-08T17:02:17.778 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:02:17.778 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T17:02:18.399 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:02:19.401 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:02:19.401 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:02:19.402 INFO:tasks.workunit.client.0.smithi172.stdout:12 2024-05-08T17:02:19.402 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-05-08T17:02:19.402 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:02:19.402 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T17:02:20.715 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:02:21.717 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:02:21.717 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:02:21.717 INFO:tasks.workunit.client.0.smithi172.stdout:13 2024-05-08T17:02:21.718 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-05-08T17:02:21.718 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:02:21.718 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T17:02:22.776 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:02:23.778 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:02:23.778 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:02:23.779 INFO:tasks.workunit.client.0.smithi172.stdout:14 2024-05-08T17:02:23.779 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-05-08T17:02:23.779 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:02:23.779 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T17:02:24.916 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:02:25.918 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:02:25.918 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:02:25.918 INFO:tasks.workunit.client.0.smithi172.stdout:15 2024-05-08T17:02:25.919 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-05-08T17:02:25.919 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:02:25.919 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T17:02:27.173 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:02:28.175 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:02:28.175 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:02:28.175 INFO:tasks.workunit.client.0.smithi172.stdout:16 2024-05-08T17:02:28.176 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-05-08T17:02:28.176 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:02:28.176 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T17:02:29.007 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:02:30.009 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:02:30.009 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:02:30.009 INFO:tasks.workunit.client.0.smithi172.stdout:17 2024-05-08T17:02:30.010 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-05-08T17:02:30.010 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:02:30.010 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T17:02:30.698 INFO:tasks.workunit.client.0.smithi172.stdout:osd.1 up in weight 1 up_from 13 up_thru 13 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/1443103358,v1:127.0.0.1:6811/1443103358] [v2:127.0.0.1:6812/1443103358,v1:127.0.0.1:6813/1443103358] exists,up a0a953bf-e981-4e54-97d5-d7c2101753a9 2024-05-08T17:02:30.702 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-08T17:02:30.702 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-08T17:02:30.702 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-08T17:02:30.702 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-05-08T17:02:30.702 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 2 2024-05-08T17:02:30.702 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-05-08T17:02:30.702 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-05-08T17:02:30.702 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-05-08T17:02:30.702 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-05-08T17:02:30.703 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/2 2024-05-08T17:02:30.703 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=4457b0f1-028b-482b-8ee3-7ee05861e6cc --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-05-08T17:02:30.703 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-08T17:02:30.703 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-05-08T17:02:30.703 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-08T17:02:30.703 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/2' 2024-05-08T17:02:30.703 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/2/journal' 2024-05-08T17:02:30.703 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-05-08T17:02:30.703 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-05-08T17:02:30.703 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-05-08T17:02:30.703 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-05-08T17:02:30.703 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-08T17:02:30.703 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-08T17:02:30.703 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-08T17:02:30.704 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T17:02:30.704 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81110 2024-05-08T17:02:30.704 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.81110/$cluster-$name.asok' 2024-05-08T17:02:30.704 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.81110/$cluster-$name.asok' 2024-05-08T17:02:30.704 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-05-08T17:02:30.704 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-05-08T17:02:30.704 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-05-08T17:02:30.704 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-05-08T17:02:30.704 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-05-08T17:02:30.704 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-08T17:02:30.704 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-08T17:02:30.704 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-08T17:02:30.704 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-08T17:02:30.705 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-05-08T17:02:30.705 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-05-08T17:02:30.705 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/2 2024-05-08T17:02:30.705 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-05-08T17:02:30.705 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=f5e9bb9e-535e-42dd-9e28-654a28c9dc8d 2024-05-08T17:02:30.706 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 f5e9bb9e-535e-42dd-9e28-654a28c9dc8d' 2024-05-08T17:02:30.706 INFO:tasks.workunit.client.0.smithi172.stdout:add osd2 f5e9bb9e-535e-42dd-9e28-654a28c9dc8d 2024-05-08T17:02:30.706 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-05-08T17:02:30.727 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAmsDtmVYm7KxAAc1DS+qNWIz+i8I5eH41Faw== 2024-05-08T17:02:30.728 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAmsDtmVYm7KxAAc1DS+qNWIz+i8I5eH41Faw=="}' 2024-05-08T17:02:30.728 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new f5e9bb9e-535e-42dd-9e28-654a28c9dc8d -i td/test-erasure-eio/2/new.json 2024-05-08T17:02:31.516 INFO:tasks.workunit.client.0.smithi172.stdout:2 2024-05-08T17:02:31.533 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/2/new.json 2024-05-08T17:02:31.534 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=4457b0f1-028b-482b-8ee3-7ee05861e6cc --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81110/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAmsDtmVYm7KxAAc1DS+qNWIz+i8I5eH41Faw== --osd-uuid f5e9bb9e-535e-42dd-9e28-654a28c9dc8d 2024-05-08T17:02:31.568 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:02:31.573+0000 7f33795a47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T17:02:31.631 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:02:31.633+0000 7f33795a47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T17:02:31.633 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:02:31.637+0000 7f33795a47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T17:02:31.633 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:02:31.637+0000 7f33795a47c0 -1 bluestore(td/test-erasure-eio/2/block) _read_bdev_label failed to open td/test-erasure-eio/2/block: (2) No such file or directory 2024-05-08T17:02:31.634 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:02:31.637+0000 7f33795a47c0 -1 bluestore(td/test-erasure-eio/2/block) _read_bdev_label failed to open td/test-erasure-eio/2/block: (2) No such file or directory 2024-05-08T17:02:31.634 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:02:31.637+0000 7f33795a47c0 -1 bluestore(td/test-erasure-eio/2/block) _read_bdev_label failed to open td/test-erasure-eio/2/block: (2) No such file or directory 2024-05-08T17:02:31.755 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:02:31.757+0000 7f33795a47c0 -1 bluestore(td/test-erasure-eio/2) _read_fsid unparsable uuid 2024-05-08T17:02:38.525 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/2/keyring 2024-05-08T17:02:38.526 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-05-08T17:02:38.526 INFO:tasks.workunit.client.0.smithi172.stdout:adding osd2 key to auth repository 2024-05-08T17:02:38.527 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-05-08T17:02:38.527 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-05-08T17:02:39.091 INFO:tasks.workunit.client.0.smithi172.stdout:start osd.2 2024-05-08T17:02:39.092 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-05-08T17:02:39.092 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=4457b0f1-028b-482b-8ee3-7ee05861e6cc --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81110/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-05-08T17:02:39.092 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-05-08T17:02:39.092 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-05-08T17:02:39.092 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-05-08T17:02:39.124 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:02:39.129+0000 7fee04d297c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T17:02:39.126 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:02:39.129+0000 7fee04d297c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T17:02:39.128 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:02:39.133+0000 7fee04d297c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T17:02:39.529 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-05-08T17:02:39.530 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-08T17:02:39.530 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-05-08T17:02:39.530 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-08T17:02:39.530 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-08T17:02:39.530 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:02:39.530 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-08T17:02:39.530 INFO:tasks.workunit.client.0.smithi172.stdout:0 2024-05-08T17:02:39.531 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:02:39.531 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T17:02:39.717 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:02:39.721+0000 7fee04d297c0 -1 Falling back to public interface 2024-05-08T17:02:39.974 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:02:40.976 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:02:40.976 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:02:40.977 INFO:tasks.workunit.client.0.smithi172.stdout:1 2024-05-08T17:02:40.977 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-08T17:02:40.978 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:02:40.978 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T17:02:41.378 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:02:42.354 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:02:42.357+0000 7fee04d297c0 -1 osd.2 0 log_to_monitors true 2024-05-08T17:02:42.379 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:02:42.379 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:02:42.379 INFO:tasks.workunit.client.0.smithi172.stdout:2 2024-05-08T17:02:42.380 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-08T17:02:42.380 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T17:02:42.380 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:02:43.189 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:02:44.191 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:02:44.191 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:02:44.191 INFO:tasks.workunit.client.0.smithi172.stdout:3 2024-05-08T17:02:44.191 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-05-08T17:02:44.192 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:02:44.192 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T17:02:44.657 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:02:45.658 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:02:45.659 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:02:45.659 INFO:tasks.workunit.client.0.smithi172.stdout:4 2024-05-08T17:02:45.659 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-05-08T17:02:45.659 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:02:45.660 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T17:02:46.230 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:02:47.231 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:02:47.231 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:02:47.232 INFO:tasks.workunit.client.0.smithi172.stdout:5 2024-05-08T17:02:47.232 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-05-08T17:02:47.232 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:02:47.232 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T17:02:47.717 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:02:48.719 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:02:48.719 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:02:48.719 INFO:tasks.workunit.client.0.smithi172.stdout:6 2024-05-08T17:02:48.720 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-05-08T17:02:48.720 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:02:48.720 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T17:02:49.359 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:02:50.360 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:02:50.361 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:02:50.361 INFO:tasks.workunit.client.0.smithi172.stdout:7 2024-05-08T17:02:50.361 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-05-08T17:02:50.361 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:02:50.362 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T17:02:50.805 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:02:51.807 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:02:51.808 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:02:51.808 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-05-08T17:02:51.808 INFO:tasks.workunit.client.0.smithi172.stdout:8 2024-05-08T17:02:51.808 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:02:51.808 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T17:02:52.444 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:02:53.446 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:02:53.446 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:02:53.447 INFO:tasks.workunit.client.0.smithi172.stdout:9 2024-05-08T17:02:53.447 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-05-08T17:02:53.447 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:02:53.447 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T17:02:53.873 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:02:54.875 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:02:54.875 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:02:54.875 INFO:tasks.workunit.client.0.smithi172.stdout:10 2024-05-08T17:02:54.876 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-05-08T17:02:54.876 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:02:54.876 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T17:02:55.368 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:02:56.370 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:02:56.370 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:02:56.370 INFO:tasks.workunit.client.0.smithi172.stdout:11 2024-05-08T17:02:56.371 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-05-08T17:02:56.371 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:02:56.371 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T17:02:56.793 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:02:57.795 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:02:57.796 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:02:57.796 INFO:tasks.workunit.client.0.smithi172.stdout:12 2024-05-08T17:02:57.797 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-05-08T17:02:57.797 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:02:57.797 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T17:02:58.461 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:02:59.463 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:02:59.463 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:02:59.464 INFO:tasks.workunit.client.0.smithi172.stdout:13 2024-05-08T17:02:59.464 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-05-08T17:02:59.464 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:02:59.464 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T17:03:00.038 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:03:01.039 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:03:01.040 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:03:01.040 INFO:tasks.workunit.client.0.smithi172.stdout:14 2024-05-08T17:03:01.040 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-05-08T17:03:01.041 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:03:01.041 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T17:03:01.879 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:03:02.881 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:03:02.881 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:03:02.881 INFO:tasks.workunit.client.0.smithi172.stdout:15 2024-05-08T17:03:02.882 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-05-08T17:03:02.882 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:03:02.882 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T17:03:03.860 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:03:04.862 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:03:04.862 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:03:04.863 INFO:tasks.workunit.client.0.smithi172.stdout:16 2024-05-08T17:03:04.863 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-05-08T17:03:04.863 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:03:04.863 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T17:03:05.545 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:03:06.547 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:03:06.547 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:03:06.547 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-05-08T17:03:06.547 INFO:tasks.workunit.client.0.smithi172.stdout:17 2024-05-08T17:03:06.548 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:03:06.548 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T17:03:07.328 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:03:08.330 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:03:08.330 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:03:08.331 INFO:tasks.workunit.client.0.smithi172.stdout:18 2024-05-08T17:03:08.331 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-05-08T17:03:08.331 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:03:08.331 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T17:03:08.828 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:03:09.830 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:03:09.830 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:03:09.831 INFO:tasks.workunit.client.0.smithi172.stdout:19 2024-05-08T17:03:09.831 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-05-08T17:03:09.831 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:03:09.831 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T17:03:10.493 INFO:tasks.workunit.client.0.smithi172.stdout:osd.2 up in weight 1 up_from 19 up_thru 19 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/2243947023,v1:127.0.0.1:6819/2243947023] [v2:127.0.0.1:6820/2243947023,v1:127.0.0.1:6821/2243947023] exists,up f5e9bb9e-535e-42dd-9e28-654a28c9dc8d 2024-05-08T17:03:10.494 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-08T17:03:10.494 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-08T17:03:10.495 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-08T17:03:10.495 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-05-08T17:03:10.495 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 3 2024-05-08T17:03:10.495 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-05-08T17:03:10.495 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-05-08T17:03:10.495 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2024-05-08T17:03:10.495 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-05-08T17:03:10.495 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/3 2024-05-08T17:03:10.495 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=4457b0f1-028b-482b-8ee3-7ee05861e6cc --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-05-08T17:03:10.495 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-08T17:03:10.495 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-05-08T17:03:10.495 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-08T17:03:10.495 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/3' 2024-05-08T17:03:10.496 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/3/journal' 2024-05-08T17:03:10.496 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-05-08T17:03:10.496 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-05-08T17:03:10.496 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-05-08T17:03:10.496 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-05-08T17:03:10.496 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-08T17:03:10.496 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-08T17:03:10.498 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-08T17:03:10.498 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T17:03:10.498 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81110 2024-05-08T17:03:10.498 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.81110/$cluster-$name.asok' 2024-05-08T17:03:10.499 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.81110/$cluster-$name.asok' 2024-05-08T17:03:10.499 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-05-08T17:03:10.499 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-05-08T17:03:10.499 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-05-08T17:03:10.499 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-05-08T17:03:10.499 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-05-08T17:03:10.499 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-08T17:03:10.499 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-08T17:03:10.499 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-08T17:03:10.499 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-08T17:03:10.499 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-05-08T17:03:10.499 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-05-08T17:03:10.499 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/3 2024-05-08T17:03:10.499 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-05-08T17:03:10.500 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=45c87850-a1b4-4b78-88b9-d95f153cb247 2024-05-08T17:03:10.500 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 45c87850-a1b4-4b78-88b9-d95f153cb247' 2024-05-08T17:03:10.500 INFO:tasks.workunit.client.0.smithi172.stdout:add osd3 45c87850-a1b4-4b78-88b9-d95f153cb247 2024-05-08T17:03:10.500 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-05-08T17:03:10.522 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBOsDtm4xF+HxAAG0HBkx12XaMgQJeZ72ev9Q== 2024-05-08T17:03:10.522 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBOsDtm4xF+HxAAG0HBkx12XaMgQJeZ72ev9Q=="}' 2024-05-08T17:03:10.522 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 45c87850-a1b4-4b78-88b9-d95f153cb247 -i td/test-erasure-eio/3/new.json 2024-05-08T17:03:11.185 INFO:tasks.workunit.client.0.smithi172.stdout:3 2024-05-08T17:03:11.251 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/3/new.json 2024-05-08T17:03:11.251 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=4457b0f1-028b-482b-8ee3-7ee05861e6cc --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81110/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBOsDtm4xF+HxAAG0HBkx12XaMgQJeZ72ev9Q== --osd-uuid 45c87850-a1b4-4b78-88b9-d95f153cb247 2024-05-08T17:03:11.251 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:03:11.240+0000 7f192ddb27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T17:03:11.279 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:03:11.280+0000 7f192ddb27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T17:03:11.280 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:03:11.284+0000 7f192ddb27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T17:03:11.280 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:03:11.284+0000 7f192ddb27c0 -1 bluestore(td/test-erasure-eio/3/block) _read_bdev_label failed to open td/test-erasure-eio/3/block: (2) No such file or directory 2024-05-08T17:03:11.281 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:03:11.284+0000 7f192ddb27c0 -1 bluestore(td/test-erasure-eio/3/block) _read_bdev_label failed to open td/test-erasure-eio/3/block: (2) No such file or directory 2024-05-08T17:03:11.281 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:03:11.284+0000 7f192ddb27c0 -1 bluestore(td/test-erasure-eio/3/block) _read_bdev_label failed to open td/test-erasure-eio/3/block: (2) No such file or directory 2024-05-08T17:03:11.410 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:03:11.412+0000 7f192ddb27c0 -1 bluestore(td/test-erasure-eio/3) _read_fsid unparsable uuid 2024-05-08T17:03:18.104 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/3/keyring 2024-05-08T17:03:18.104 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-05-08T17:03:18.105 INFO:tasks.workunit.client.0.smithi172.stdout:adding osd3 key to auth repository 2024-05-08T17:03:18.106 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2024-05-08T17:03:18.106 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-05-08T17:03:18.652 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2024-05-08T17:03:18.653 INFO:tasks.workunit.client.0.smithi172.stdout:start osd.3 2024-05-08T17:03:18.653 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-05-08T17:03:18.653 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=4457b0f1-028b-482b-8ee3-7ee05861e6cc --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81110/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-05-08T17:03:18.653 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-05-08T17:03:18.654 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-05-08T17:03:18.684 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:03:18.688+0000 7fbab61037c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T17:03:18.686 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:03:18.688+0000 7fbab61037c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T17:03:18.687 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:03:18.692+0000 7fbab61037c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T17:03:19.052 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2024-05-08T17:03:19.052 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-08T17:03:19.052 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-05-08T17:03:19.052 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-08T17:03:19.052 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-08T17:03:19.053 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:03:19.053 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-08T17:03:19.053 INFO:tasks.workunit.client.0.smithi172.stdout:0 2024-05-08T17:03:19.053 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T17:03:19.053 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:03:19.237 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:03:19.240+0000 7fbab61037c0 -1 Falling back to public interface 2024-05-08T17:03:19.487 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:03:20.488 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:03:20.489 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:03:20.489 INFO:tasks.workunit.client.0.smithi172.stdout:1 2024-05-08T17:03:20.489 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-08T17:03:20.489 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:03:20.490 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T17:03:20.898 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:03:21.900 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:03:21.901 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:03:21.901 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-08T17:03:21.901 INFO:tasks.workunit.client.0.smithi172.stdout:2 2024-05-08T17:03:21.901 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:03:21.901 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T17:03:22.057 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:03:22.060+0000 7fbab61037c0 -1 osd.3 0 log_to_monitors true 2024-05-08T17:03:22.427 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:03:23.428 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:03:23.429 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:03:23.429 INFO:tasks.workunit.client.0.smithi172.stdout:3 2024-05-08T17:03:23.429 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-05-08T17:03:23.429 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:03:23.430 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T17:03:24.101 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:03:25.103 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:03:25.103 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:03:25.103 INFO:tasks.workunit.client.0.smithi172.stdout:4 2024-05-08T17:03:25.104 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-05-08T17:03:25.104 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:03:25.104 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T17:03:25.715 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:03:26.718 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:03:26.718 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:03:26.718 INFO:tasks.workunit.client.0.smithi172.stdout:5 2024-05-08T17:03:26.719 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-05-08T17:03:26.719 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:03:26.719 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T17:03:27.146 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:03:28.148 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:03:28.148 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:03:28.148 INFO:tasks.workunit.client.0.smithi172.stdout:6 2024-05-08T17:03:28.149 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-05-08T17:03:28.149 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:03:28.149 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T17:03:28.623 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:03:29.626 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:03:29.626 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:03:29.626 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-05-08T17:03:29.627 INFO:tasks.workunit.client.0.smithi172.stdout:7 2024-05-08T17:03:29.627 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:03:29.627 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T17:03:30.107 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:03:31.108 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:03:31.108 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:03:31.109 INFO:tasks.workunit.client.0.smithi172.stdout:8 2024-05-08T17:03:31.109 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-05-08T17:03:31.109 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:03:31.109 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T17:03:31.653 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:03:32.654 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:03:32.655 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:03:32.655 INFO:tasks.workunit.client.0.smithi172.stdout:9 2024-05-08T17:03:32.655 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-05-08T17:03:32.655 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:03:32.656 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T17:03:33.094 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:03:34.095 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:03:34.095 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:03:34.096 INFO:tasks.workunit.client.0.smithi172.stdout:10 2024-05-08T17:03:34.096 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-05-08T17:03:34.096 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:03:34.096 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T17:03:34.567 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:03:35.569 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:03:35.569 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:03:35.570 INFO:tasks.workunit.client.0.smithi172.stdout:11 2024-05-08T17:03:35.570 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-05-08T17:03:35.570 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:03:35.570 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T17:03:36.130 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:03:37.133 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:03:37.133 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:03:37.133 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-05-08T17:03:37.133 INFO:tasks.workunit.client.0.smithi172.stdout:12 2024-05-08T17:03:37.134 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:03:37.134 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T17:03:37.696 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:03:38.698 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:03:38.698 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:03:38.699 INFO:tasks.workunit.client.0.smithi172.stdout:13 2024-05-08T17:03:38.699 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-05-08T17:03:38.699 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:03:38.699 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T17:03:39.887 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:03:40.884 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:03:40.885 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:03:40.885 INFO:tasks.workunit.client.0.smithi172.stdout:14 2024-05-08T17:03:40.886 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-05-08T17:03:40.886 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:03:40.886 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T17:03:41.654 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:03:42.656 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:03:42.657 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:03:42.657 INFO:tasks.workunit.client.0.smithi172.stdout:15 2024-05-08T17:03:42.657 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-05-08T17:03:42.657 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:03:42.658 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T17:03:43.715 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:03:44.716 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:03:44.717 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:03:44.717 INFO:tasks.workunit.client.0.smithi172.stdout:16 2024-05-08T17:03:44.717 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-05-08T17:03:44.718 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:03:44.718 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T17:03:45.577 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:03:46.579 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:03:46.579 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:03:46.579 INFO:tasks.workunit.client.0.smithi172.stdout:17 2024-05-08T17:03:46.580 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-05-08T17:03:46.580 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:03:46.580 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T17:03:47.433 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:03:48.435 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:03:48.435 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:03:48.435 INFO:tasks.workunit.client.0.smithi172.stdout:18 2024-05-08T17:03:48.436 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-05-08T17:03:48.436 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:03:48.436 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T17:03:49.243 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:03:50.244 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:03:50.245 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:03:50.245 INFO:tasks.workunit.client.0.smithi172.stdout:19 2024-05-08T17:03:50.245 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-05-08T17:03:50.245 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:03:50.246 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T17:03:50.722 INFO:tasks.workunit.client.0.smithi172.stdout:osd.3 up in weight 1 up_from 28 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/2645133208,v1:127.0.0.1:6827/2645133208] [v2:127.0.0.1:6828/2645133208,v1:127.0.0.1:6829/2645133208] exists,up 45c87850-a1b4-4b78-88b9-d95f153cb247 2024-05-08T17:03:50.723 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-08T17:03:50.723 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-08T17:03:50.723 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-08T17:03:50.724 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: get_asok_path osd.0 2024-05-08T17:03:50.724 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-05-08T17:03:50.724 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-05-08T17:03:50.725 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-08T17:03:50.725 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T17:03:50.725 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81110 2024-05-08T17:03:50.725 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.81110/ceph-osd.0.asok 2024-05-08T17:03:50.725 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: CEPH_ARGS= 2024-05-08T17:03:50.726 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: ceph --admin-daemon /tmp/ceph-asok.81110/ceph-osd.0.asok log flush 2024-05-08T17:03:50.836 INFO:tasks.workunit.client.0.smithi172.stdout:{} 2024-05-08T17:03:50.847 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:56: setup_osds: grep 'load: jerasure.*lrc' td/test-erasure-eio/osd.0.log 2024-05-08T17:03:50.853 INFO:tasks.workunit.client.0.smithi172.stdout:2024-05-08T17:01:22.778+0000 7f1240bc27c0 0 load: jerasure load: lrc 2024-05-08T17:03:50.854 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:319: TEST_rados_get_bad_size_shard_1: local poolname=pool-jerasure 2024-05-08T17:03:50.854 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:320: TEST_rados_get_bad_size_shard_1: create_erasure_coded_pool pool-jerasure 2 1 2024-05-08T17:03:50.855 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:67: create_erasure_coded_pool: local poolname=pool-jerasure 2024-05-08T17:03:50.855 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:68: create_erasure_coded_pool: shift 2024-05-08T17:03:50.855 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:69: create_erasure_coded_pool: local k=2 2024-05-08T17:03:50.855 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:70: create_erasure_coded_pool: shift 2024-05-08T17:03:50.855 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:71: create_erasure_coded_pool: local m=1 2024-05-08T17:03:50.855 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:72: create_erasure_coded_pool: shift 2024-05-08T17:03:50.855 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:74: create_erasure_coded_pool: ceph osd erasure-code-profile set myprofile plugin=jerasure k=2 m=1 crush-failure-domain=osd 2024-05-08T17:03:51.946 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:78: create_erasure_coded_pool: create_pool pool-jerasure 1 1 erasure myprofile 2024-05-08T17:03:51.947 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create pool-jerasure 1 1 erasure myprofile 2024-05-08T17:03:52.910 INFO:tasks.workunit.client.0.smithi172.stderr:pool 'pool-jerasure' already exists 2024-05-08T17:03:52.928 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-05-08T17:03:53.930 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:80: create_erasure_coded_pool: wait_for_clean 2024-05-08T17:03:53.930 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-05-08T17:03:53.930 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-05-08T17:03:53.930 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-05-08T17:03:53.930 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-05-08T17:03:53.931 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-05-08T17:03:53.931 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-05-08T17:03:53.932 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-05-08T17:03:53.932 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-05-08T17:03:53.932 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-05-08T17:03:54.060 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-05-08T17:03:54.060 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-05-08T17:03:54.060 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-05-08T17:03:54.060 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-05-08T17:03:54.060 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-05-08T17:03:54.060 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-05-08T17:03:54.532 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-05-08T17:03:54.532 INFO:tasks.workunit.client.0.smithi172.stderr:1 2024-05-08T17:03:54.533 INFO:tasks.workunit.client.0.smithi172.stderr:2 2024-05-08T17:03:54.533 INFO:tasks.workunit.client.0.smithi172.stderr:3' 2024-05-08T17:03:54.533 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-05-08T17:03:54.533 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T17:03:54.533 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-05-08T17:03:54.698 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=25769803808 2024-05-08T17:03:54.698 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 25769803808 2024-05-08T17:03:54.698 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803808' 2024-05-08T17:03:54.698 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T17:03:54.698 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-05-08T17:03:54.848 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=55834574872 2024-05-08T17:03:54.849 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 55834574872 2024-05-08T17:03:54.849 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803808 1-55834574872' 2024-05-08T17:03:54.849 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T17:03:54.849 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-05-08T17:03:54.997 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=81604378637 2024-05-08T17:03:54.997 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 81604378637 2024-05-08T17:03:54.998 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803808 1-55834574872 2-81604378637' 2024-05-08T17:03:54.998 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T17:03:54.998 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-05-08T17:03:55.146 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=120259084294 2024-05-08T17:03:55.146 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 120259084294 2024-05-08T17:03:55.146 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803808 1-55834574872 2-81604378637 3-120259084294' 2024-05-08T17:03:55.146 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-08T17:03:55.146 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-25769803808 2024-05-08T17:03:55.146 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-08T17:03:55.147 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-05-08T17:03:55.147 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-25769803808 2024-05-08T17:03:55.147 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-08T17:03:55.148 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=25769803808 2024-05-08T17:03:55.148 INFO:tasks.workunit.client.0.smithi172.stdout:waiting osd.0 seq 25769803808 2024-05-08T17:03:55.149 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 25769803808' 2024-05-08T17:03:55.149 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-05-08T17:03:55.559 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 25769803808 -lt 25769803808 2024-05-08T17:03:55.559 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-08T17:03:55.560 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-55834574872 2024-05-08T17:03:55.560 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-08T17:03:55.561 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-05-08T17:03:55.562 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-55834574872 2024-05-08T17:03:55.562 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-08T17:03:55.563 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=55834574872 2024-05-08T17:03:55.564 INFO:tasks.workunit.client.0.smithi172.stdout:waiting osd.1 seq 55834574872 2024-05-08T17:03:55.564 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 55834574872' 2024-05-08T17:03:55.564 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-05-08T17:03:55.975 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 55834574873 -lt 55834574872 2024-05-08T17:03:55.976 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-08T17:03:55.977 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-81604378637 2024-05-08T17:03:55.977 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-08T17:03:55.979 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-05-08T17:03:55.980 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-81604378637 2024-05-08T17:03:55.980 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-08T17:03:55.982 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=81604378637 2024-05-08T17:03:55.982 INFO:tasks.workunit.client.0.smithi172.stdout:waiting osd.2 seq 81604378637 2024-05-08T17:03:55.983 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 81604378637' 2024-05-08T17:03:55.983 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-05-08T17:03:56.371 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 81604378637 -lt 81604378637 2024-05-08T17:03:56.372 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-08T17:03:56.373 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-08T17:03:56.373 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-120259084294 2024-05-08T17:03:56.374 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-05-08T17:03:56.374 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-120259084294 2024-05-08T17:03:56.375 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-08T17:03:56.376 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=120259084294 2024-05-08T17:03:56.376 INFO:tasks.workunit.client.0.smithi172.stdout:waiting osd.3 seq 120259084294 2024-05-08T17:03:56.377 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 120259084294' 2024-05-08T17:03:56.377 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-05-08T17:03:56.801 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 120259084294 -lt 120259084294 2024-05-08T17:03:56.802 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-05-08T17:03:56.802 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-05-08T17:03:56.802 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-05-08T17:03:57.360 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-05-08T17:03:57.360 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-05-08T17:03:57.360 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-05-08T17:03:57.360 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-05-08T17:03:57.360 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-05-08T17:03:57.360 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-05-08T17:03:57.360 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-05-08T17:03:57.360 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-05-08T17:03:57.798 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-05-08T17:03:57.798 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-05-08T17:03:57.799 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-05-08T17:03:57.799 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-05-08T17:03:58.354 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-05-08T17:03:58.354 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-05-08T17:03:58.354 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-05-08T17:03:58.354 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:322: TEST_rados_get_bad_size_shard_1: local shard_id=1 2024-05-08T17:03:58.354 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:323: TEST_rados_get_bad_size_shard_1: rados_get_data_bad_size td/test-erasure-eio 1 10 2024-05-08T17:03:58.354 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:222: rados_get_data_bad_size: local dir=td/test-erasure-eio 2024-05-08T17:03:58.354 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:223: rados_get_data_bad_size: shift 2024-05-08T17:03:58.354 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:224: rados_get_data_bad_size: local shard_id=1 2024-05-08T17:03:58.355 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:225: rados_get_data_bad_size: shift 2024-05-08T17:03:58.355 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:226: rados_get_data_bad_size: local bytes=10 2024-05-08T17:03:58.355 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:227: rados_get_data_bad_size: shift 2024-05-08T17:03:58.355 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:228: rados_get_data_bad_size: local mode=set 2024-05-08T17:03:58.355 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:230: rados_get_data_bad_size: local poolname=pool-jerasure 2024-05-08T17:03:58.355 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:231: rados_get_data_bad_size: local objname=obj-size-81110-1-10 2024-05-08T17:03:58.355 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:232: rados_get_data_bad_size: rados_put td/test-erasure-eio pool-jerasure obj-size-81110-1-10 2024-05-08T17:03:58.355 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:90: rados_put: local dir=td/test-erasure-eio 2024-05-08T17:03:58.355 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:91: rados_put: local poolname=pool-jerasure 2024-05-08T17:03:58.355 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:92: rados_put: local objname=obj-size-81110-1-10 2024-05-08T17:03:58.355 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-05-08T17:03:58.355 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 AAA 2024-05-08T17:03:58.356 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-05-08T17:03:58.356 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 BBB 2024-05-08T17:03:58.356 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-05-08T17:03:58.356 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 CCCC 2024-05-08T17:03:58.356 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-05-08T17:03:58.356 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 DDDD 2024-05-08T17:03:58.356 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:100: rados_put: rados --pool pool-jerasure put obj-size-81110-1-10 td/test-erasure-eio/ORIGINAL 2024-05-08T17:03:58.542 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:236: rados_get_data_bad_size: set_size obj-size-81110-1-10 td/test-erasure-eio 1 10 set 2024-05-08T17:03:58.542 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:192: set_size: local objname=obj-size-81110-1-10 2024-05-08T17:03:58.542 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:193: set_size: shift 2024-05-08T17:03:58.542 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:194: set_size: local dir=td/test-erasure-eio 2024-05-08T17:03:58.542 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:195: set_size: shift 2024-05-08T17:03:58.543 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:196: set_size: local shard_id=1 2024-05-08T17:03:58.543 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:197: set_size: shift 2024-05-08T17:03:58.543 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:198: set_size: local bytes=10 2024-05-08T17:03:58.543 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:199: set_size: shift 2024-05-08T17:03:58.543 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:200: set_size: local mode=set 2024-05-08T17:03:58.543 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:202: set_size: local poolname=pool-jerasure 2024-05-08T17:03:58.543 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: get_osds pool-jerasure obj-size-81110-1-10 2024-05-08T17:03:58.543 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2024-05-08T17:03:58.543 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-size-81110-1-10 2024-05-08T17:03:58.543 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure obj-size-81110-1-10 2024-05-08T17:03:58.544 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-05-08T17:03:58.968 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-05-08T17:03:58.968 INFO:tasks.workunit.client.0.smithi172.stderr:1 2024-05-08T17:03:58.968 INFO:tasks.workunit.client.0.smithi172.stderr:2' 2024-05-08T17:03:58.968 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 2 2024-05-08T17:03:58.969 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: initial_osds=('3' '1' '2') 2024-05-08T17:03:58.969 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: local -a initial_osds 2024-05-08T17:03:58.969 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:204: set_size: local osd_id=1 2024-05-08T17:03:58.969 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:205: set_size: ceph osd set noout 2024-05-08T17:03:59.750 INFO:tasks.workunit.client.0.smithi172.stderr:noout is set 2024-05-08T17:03:59.773 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:206: set_size: '[' set = add ']' 2024-05-08T17:03:59.773 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:210: set_size: '[' 10 = 0 ']' 2024-05-08T17:03:59.773 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:214: set_size: dd if=/dev/urandom bs=10 count=1 of=td/test-erasure-eio/CORRUPT 2024-05-08T17:03:59.776 INFO:tasks.workunit.client.0.smithi172.stderr:1+0 records in 2024-05-08T17:03:59.776 INFO:tasks.workunit.client.0.smithi172.stderr:1+0 records out 2024-05-08T17:03:59.776 INFO:tasks.workunit.client.0.smithi172.stderr:10 bytes copied, 0.000117977 s, 84.8 kB/s 2024-05-08T17:03:59.776 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:216: set_size: objectstore_tool td/test-erasure-eio 1 obj-size-81110-1-10 set-bytes td/test-erasure-eio/CORRUPT 2024-05-08T17:03:59.776 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/test-erasure-eio 2024-05-08T17:03:59.776 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-05-08T17:03:59.776 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-05-08T17:03:59.777 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-05-08T17:03:59.777 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/test-erasure-eio 1 obj-size-81110-1-10 set-bytes td/test-erasure-eio/CORRUPT 2024-05-08T17:03:59.777 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/test-erasure-eio 2024-05-08T17:03:59.777 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-05-08T17:03:59.777 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-05-08T17:03:59.777 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-05-08T17:03:59.777 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/test-erasure-eio TERM osd.1 2024-05-08T17:03:59.777 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-08T17:03:59.777 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-08T17:03:59.777 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-08T17:03:59.778 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-08T17:03:59.778 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-08T17:04:01.090 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-08T17:04:01.090 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/test-erasure-eio 1 obj-size-81110-1-10 set-bytes td/test-erasure-eio/CORRUPT 2024-05-08T17:04:01.090 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/test-erasure-eio 2024-05-08T17:04:01.090 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-05-08T17:04:01.090 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-05-08T17:04:01.091 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-05-08T17:04:01.091 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/test-erasure-eio/1 2024-05-08T17:04:01.091 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/test-erasure-eio/1 obj-size-81110-1-10 set-bytes td/test-erasure-eio/CORRUPT 2024-05-08T17:04:03.383 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/test-erasure-eio 1 2024-05-08T17:04:03.383 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/test-erasure-eio 2024-05-08T17:04:03.383 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-05-08T17:04:03.384 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-05-08T17:04:03.384 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-05-08T17:04:03.384 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/test-erasure-eio/1 2024-05-08T17:04:03.384 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=4457b0f1-028b-482b-8ee3-7ee05861e6cc --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-05-08T17:04:03.384 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-08T17:04:03.384 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-05-08T17:04:03.384 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-08T17:04:03.384 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/1' 2024-05-08T17:04:03.384 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/1/journal' 2024-05-08T17:04:03.384 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-05-08T17:04:03.384 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-05-08T17:04:03.384 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-05-08T17:04:03.384 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-05-08T17:04:03.384 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-08T17:04:03.384 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-08T17:04:03.386 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-08T17:04:03.386 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T17:04:03.386 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81110 2024-05-08T17:04:03.386 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.81110/$cluster-$name.asok' 2024-05-08T17:04:03.387 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.81110/$cluster-$name.asok' 2024-05-08T17:04:03.387 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-05-08T17:04:03.387 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-05-08T17:04:03.387 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-05-08T17:04:03.387 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-08T17:04:03.387 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-08T17:04:03.387 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-08T17:04:03.387 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-08T17:04:03.387 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-05-08T17:04:03.387 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-05-08T17:04:03.387 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/test-erasure-eio/1 2024-05-08T17:04:03.388 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-05-08T17:04:03.388 INFO:tasks.workunit.client.0.smithi172.stderr:start osd.1 2024-05-08T17:04:03.388 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=4457b0f1-028b-482b-8ee3-7ee05861e6cc --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81110/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-05-08T17:04:03.388 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/test-erasure-eio/1/whoami 2024-05-08T17:04:03.388 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-05-08T17:04:03.389 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-05-08T17:04:03.389 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-05-08T17:04:03.389 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-05-08T17:04:03.421 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:04:03.427+0000 7fe6d71027c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T17:04:03.433 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:04:03.439+0000 7fe6d71027c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T17:04:03.434 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:04:03.439+0000 7fe6d71027c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T17:04:03.815 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-05-08T17:04:03.815 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-08T17:04:03.815 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-05-08T17:04:03.815 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-08T17:04:03.815 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-08T17:04:03.815 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:04:03.815 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-08T17:04:03.815 INFO:tasks.workunit.client.0.smithi172.stderr:0 2024-05-08T17:04:03.815 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:04:03.815 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T17:04:03.992 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:04:03.999+0000 7fe6d71027c0 -1 Falling back to public interface 2024-05-08T17:04:04.244 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:04:05.246 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:04:05.246 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:04:05.246 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-08T17:04:05.246 INFO:tasks.workunit.client.0.smithi172.stderr:1 2024-05-08T17:04:05.247 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:04:05.247 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T17:04:05.706 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:04:06.707 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:04:06.707 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:04:06.707 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-08T17:04:06.707 INFO:tasks.workunit.client.0.smithi172.stderr:2 2024-05-08T17:04:06.708 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:04:06.708 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T17:04:06.753 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:04:06.759+0000 7fe6d71027c0 -1 osd.1 36 log_to_monitors true 2024-05-08T17:04:07.201 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:04:08.203 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:04:08.204 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:04:08.204 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-05-08T17:04:08.204 INFO:tasks.workunit.client.0.smithi172.stderr:3 2024-05-08T17:04:08.204 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:04:08.204 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T17:04:08.809 INFO:tasks.workunit.client.0.smithi172.stderr:osd.1 up in weight 1 up_from 43 up_thru 43 down_at 39 last_clean_interval [13,36) [v2:127.0.0.1:6810/2842648639,v1:127.0.0.1:6811/2842648639] [v2:127.0.0.1:6812/2842648639,v1:127.0.0.1:6813/2842648639] exists,up a0a953bf-e981-4e54-97d5-d7c2101753a9 2024-05-08T17:04:08.809 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-08T17:04:08.810 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-08T17:04:08.810 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-08T17:04:08.810 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-05-08T17:04:08.810 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-05-08T17:04:08.810 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-05-08T17:04:08.810 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-05-08T17:04:08.810 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-05-08T17:04:08.811 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-05-08T17:04:08.811 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-05-08T17:04:08.812 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-05-08T17:04:08.812 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-05-08T17:04:08.812 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-05-08T17:04:08.917 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-05-08T17:04:08.917 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-05-08T17:04:08.917 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-05-08T17:04:08.917 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-05-08T17:04:08.917 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-05-08T17:04:08.918 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-05-08T17:04:09.326 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-05-08T17:04:09.326 INFO:tasks.workunit.client.0.smithi172.stderr:1 2024-05-08T17:04:09.326 INFO:tasks.workunit.client.0.smithi172.stderr:2 2024-05-08T17:04:09.326 INFO:tasks.workunit.client.0.smithi172.stderr:3' 2024-05-08T17:04:09.326 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-05-08T17:04:09.326 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T17:04:09.326 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-05-08T17:04:09.505 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=25769803813 2024-05-08T17:04:09.505 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 25769803813 2024-05-08T17:04:09.505 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803813' 2024-05-08T17:04:09.505 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T17:04:09.505 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-05-08T17:04:09.646 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=184683593731 2024-05-08T17:04:09.646 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 184683593731 2024-05-08T17:04:09.646 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803813 1-184683593731' 2024-05-08T17:04:09.646 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T17:04:09.646 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-05-08T17:04:09.788 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=81604378641 2024-05-08T17:04:09.789 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 81604378641 2024-05-08T17:04:09.789 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803813 1-184683593731 2-81604378641' 2024-05-08T17:04:09.789 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T17:04:09.789 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-05-08T17:04:09.968 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=120259084299 2024-05-08T17:04:09.969 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 120259084299 2024-05-08T17:04:09.969 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803813 1-184683593731 2-81604378641 3-120259084299' 2024-05-08T17:04:09.969 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-08T17:04:09.969 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-25769803813 2024-05-08T17:04:09.969 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-08T17:04:09.970 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-05-08T17:04:09.971 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-25769803813 2024-05-08T17:04:09.971 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-08T17:04:09.972 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=25769803813 2024-05-08T17:04:09.972 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 25769803813' 2024-05-08T17:04:09.972 INFO:tasks.workunit.client.0.smithi172.stderr:waiting osd.0 seq 25769803813 2024-05-08T17:04:09.973 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-05-08T17:04:10.362 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 25769803811 -lt 25769803813 2024-05-08T17:04:10.362 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-05-08T17:04:11.363 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-05-08T17:04:11.364 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-05-08T17:04:11.785 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 25769803813 -lt 25769803813 2024-05-08T17:04:11.785 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-08T17:04:11.786 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-184683593731 2024-05-08T17:04:11.787 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-08T17:04:11.788 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-05-08T17:04:11.789 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-184683593731 2024-05-08T17:04:11.789 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-08T17:04:11.791 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=184683593731 2024-05-08T17:04:11.791 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 184683593731' 2024-05-08T17:04:11.791 INFO:tasks.workunit.client.0.smithi172.stderr:waiting osd.1 seq 184683593731 2024-05-08T17:04:11.791 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-05-08T17:04:12.189 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 184683593731 -lt 184683593731 2024-05-08T17:04:12.189 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-08T17:04:12.190 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-81604378641 2024-05-08T17:04:12.191 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-08T17:04:12.192 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-05-08T17:04:12.193 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-81604378641 2024-05-08T17:04:12.193 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-08T17:04:12.194 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=81604378641 2024-05-08T17:04:12.194 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 81604378641' 2024-05-08T17:04:12.194 INFO:tasks.workunit.client.0.smithi172.stderr:waiting osd.2 seq 81604378641 2024-05-08T17:04:12.195 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-05-08T17:04:12.637 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 81604378642 -lt 81604378641 2024-05-08T17:04:12.637 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-08T17:04:12.638 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-08T17:04:12.638 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-120259084299 2024-05-08T17:04:12.640 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-05-08T17:04:12.642 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-120259084299 2024-05-08T17:04:12.642 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-08T17:04:12.642 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=120259084299 2024-05-08T17:04:12.642 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 120259084299' 2024-05-08T17:04:12.642 INFO:tasks.workunit.client.0.smithi172.stderr:waiting osd.3 seq 120259084299 2024-05-08T17:04:12.643 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-05-08T17:04:13.079 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 120259084300 -lt 120259084299 2024-05-08T17:04:13.080 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-05-08T17:04:13.081 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-05-08T17:04:13.081 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-05-08T17:04:13.642 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-05-08T17:04:13.643 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-05-08T17:04:13.643 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-05-08T17:04:13.644 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-05-08T17:04:13.644 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-05-08T17:04:13.644 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-05-08T17:04:13.644 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-05-08T17:04:13.644 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-05-08T17:04:14.072 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-05-08T17:04:14.073 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-05-08T17:04:14.073 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-05-08T17:04:14.073 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-05-08T17:04:14.585 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-05-08T17:04:14.586 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-05-08T17:04:14.586 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-05-08T17:04:14.586 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:217: set_size: rm -f td/test-erasure-eio/CORRUPT 2024-05-08T17:04:14.586 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:218: set_size: ceph osd unset noout 2024-05-08T17:04:15.403 INFO:tasks.workunit.client.0.smithi172.stderr:noout is unset 2024-05-08T17:04:15.427 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:238: rados_get_data_bad_size: rados_get td/test-erasure-eio pool-jerasure obj-size-81110-1-10 2024-05-08T17:04:15.427 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2024-05-08T17:04:15.427 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2024-05-08T17:04:15.427 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=obj-size-81110-1-10 2024-05-08T17:04:15.427 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=ok 2024-05-08T17:04:15.427 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' ok = fail ']' 2024-05-08T17:04:15.427 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:120: rados_get: rados --pool pool-jerasure get obj-size-81110-1-10 td/test-erasure-eio/COPY 2024-05-08T17:04:15.626 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:121: rados_get: diff td/test-erasure-eio/ORIGINAL td/test-erasure-eio/COPY 2024-05-08T17:04:15.627 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:122: rados_get: rm td/test-erasure-eio/COPY 2024-05-08T17:04:15.630 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:241: rados_get_data_bad_size: expr 1 + 1 2024-05-08T17:04:15.632 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:241: rados_get_data_bad_size: shard_id=2 2024-05-08T17:04:15.632 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:242: rados_get_data_bad_size: set_size obj-size-81110-1-10 td/test-erasure-eio 2 10 set 2024-05-08T17:04:15.632 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:192: set_size: local objname=obj-size-81110-1-10 2024-05-08T17:04:15.632 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:193: set_size: shift 2024-05-08T17:04:15.632 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:194: set_size: local dir=td/test-erasure-eio 2024-05-08T17:04:15.633 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:195: set_size: shift 2024-05-08T17:04:15.633 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:196: set_size: local shard_id=2 2024-05-08T17:04:15.633 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:197: set_size: shift 2024-05-08T17:04:15.633 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:198: set_size: local bytes=10 2024-05-08T17:04:15.633 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:199: set_size: shift 2024-05-08T17:04:15.633 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:200: set_size: local mode=set 2024-05-08T17:04:15.633 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:202: set_size: local poolname=pool-jerasure 2024-05-08T17:04:15.633 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: get_osds pool-jerasure obj-size-81110-1-10 2024-05-08T17:04:15.633 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2024-05-08T17:04:15.633 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-size-81110-1-10 2024-05-08T17:04:15.633 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure obj-size-81110-1-10 2024-05-08T17:04:15.634 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-05-08T17:04:16.072 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-05-08T17:04:16.073 INFO:tasks.workunit.client.0.smithi172.stderr:1 2024-05-08T17:04:16.073 INFO:tasks.workunit.client.0.smithi172.stderr:2' 2024-05-08T17:04:16.073 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 2 2024-05-08T17:04:16.073 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: initial_osds=('3' '1' '2') 2024-05-08T17:04:16.073 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: local -a initial_osds 2024-05-08T17:04:16.073 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:204: set_size: local osd_id=2 2024-05-08T17:04:16.073 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:205: set_size: ceph osd set noout 2024-05-08T17:04:16.750 INFO:tasks.workunit.client.0.smithi172.stderr:noout is set 2024-05-08T17:04:16.771 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:206: set_size: '[' set = add ']' 2024-05-08T17:04:16.771 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:210: set_size: '[' 10 = 0 ']' 2024-05-08T17:04:16.771 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:214: set_size: dd if=/dev/urandom bs=10 count=1 of=td/test-erasure-eio/CORRUPT 2024-05-08T17:04:16.772 INFO:tasks.workunit.client.0.smithi172.stderr:1+0 records in 2024-05-08T17:04:16.772 INFO:tasks.workunit.client.0.smithi172.stderr:1+0 records out 2024-05-08T17:04:16.772 INFO:tasks.workunit.client.0.smithi172.stderr:10 bytes copied, 0.00017318 s, 57.7 kB/s 2024-05-08T17:04:16.774 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:216: set_size: objectstore_tool td/test-erasure-eio 2 obj-size-81110-1-10 set-bytes td/test-erasure-eio/CORRUPT 2024-05-08T17:04:16.774 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/test-erasure-eio 2024-05-08T17:04:16.774 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-05-08T17:04:16.774 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=2 2024-05-08T17:04:16.774 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-05-08T17:04:16.774 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/test-erasure-eio 2 obj-size-81110-1-10 set-bytes td/test-erasure-eio/CORRUPT 2024-05-08T17:04:16.774 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/test-erasure-eio 2024-05-08T17:04:16.774 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-05-08T17:04:16.774 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=2 2024-05-08T17:04:16.774 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-05-08T17:04:16.774 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/test-erasure-eio TERM osd.2 2024-05-08T17:04:16.775 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-08T17:04:16.775 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-08T17:04:16.775 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-08T17:04:16.775 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-08T17:04:16.775 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-08T17:04:18.087 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-08T17:04:18.087 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/test-erasure-eio 2 obj-size-81110-1-10 set-bytes td/test-erasure-eio/CORRUPT 2024-05-08T17:04:18.088 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/test-erasure-eio 2024-05-08T17:04:18.088 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-05-08T17:04:18.088 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=2 2024-05-08T17:04:18.088 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-05-08T17:04:18.088 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/test-erasure-eio/2 2024-05-08T17:04:18.088 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/test-erasure-eio/2 obj-size-81110-1-10 set-bytes td/test-erasure-eio/CORRUPT 2024-05-08T17:04:20.237 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/test-erasure-eio 2 2024-05-08T17:04:20.237 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/test-erasure-eio 2024-05-08T17:04:20.237 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-05-08T17:04:20.238 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-05-08T17:04:20.238 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-05-08T17:04:20.238 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/test-erasure-eio/2 2024-05-08T17:04:20.238 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=4457b0f1-028b-482b-8ee3-7ee05861e6cc --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-05-08T17:04:20.238 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-08T17:04:20.238 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-05-08T17:04:20.238 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-08T17:04:20.238 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/2' 2024-05-08T17:04:20.238 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/2/journal' 2024-05-08T17:04:20.238 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-05-08T17:04:20.238 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-05-08T17:04:20.238 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-05-08T17:04:20.238 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-05-08T17:04:20.239 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-08T17:04:20.239 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-08T17:04:20.239 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-08T17:04:20.239 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T17:04:20.239 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81110 2024-05-08T17:04:20.239 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.81110/$cluster-$name.asok' 2024-05-08T17:04:20.240 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.81110/$cluster-$name.asok' 2024-05-08T17:04:20.240 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-05-08T17:04:20.240 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-05-08T17:04:20.240 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-05-08T17:04:20.240 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-08T17:04:20.240 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-08T17:04:20.240 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-08T17:04:20.240 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-08T17:04:20.241 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-05-08T17:04:20.241 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-05-08T17:04:20.241 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/test-erasure-eio/2 2024-05-08T17:04:20.241 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-05-08T17:04:20.241 INFO:tasks.workunit.client.0.smithi172.stderr:start osd.2 2024-05-08T17:04:20.242 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=4457b0f1-028b-482b-8ee3-7ee05861e6cc --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81110/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-05-08T17:04:20.242 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/test-erasure-eio/2/whoami 2024-05-08T17:04:20.243 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-05-08T17:04:20.243 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-05-08T17:04:20.243 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-05-08T17:04:20.244 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-05-08T17:04:20.274 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:04:20.279+0000 7fccd00c17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T17:04:20.277 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:04:20.283+0000 7fccd00c17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T17:04:20.278 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:04:20.283+0000 7fccd00c17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T17:04:20.661 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-05-08T17:04:20.662 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-08T17:04:20.662 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-05-08T17:04:20.662 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-08T17:04:20.662 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-08T17:04:20.662 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:04:20.662 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-08T17:04:20.662 INFO:tasks.workunit.client.0.smithi172.stderr:0 2024-05-08T17:04:20.662 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:04:20.662 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T17:04:20.845 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:04:20.851+0000 7fccd00c17c0 -1 Falling back to public interface 2024-05-08T17:04:21.068 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:04:22.069 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:04:22.069 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:04:22.069 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-08T17:04:22.069 INFO:tasks.workunit.client.0.smithi172.stderr:1 2024-05-08T17:04:22.070 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:04:22.070 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T17:04:22.494 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:04:23.495 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:04:23.496 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:04:23.496 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-08T17:04:23.496 INFO:tasks.workunit.client.0.smithi172.stderr:2 2024-05-08T17:04:23.496 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:04:23.496 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T17:04:23.529 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:04:23.535+0000 7fccd00c17c0 -1 osd.2 47 log_to_monitors true 2024-05-08T17:04:23.931 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:04:24.932 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:04:24.932 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:04:24.932 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-05-08T17:04:24.933 INFO:tasks.workunit.client.0.smithi172.stderr:3 2024-05-08T17:04:24.933 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:04:24.933 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T17:04:25.420 INFO:tasks.workunit.client.0.smithi172.stderr:osd.2 up in weight 1 up_from 53 up_thru 39 down_at 50 last_clean_interval [19,47) [v2:127.0.0.1:6818/709392629,v1:127.0.0.1:6819/709392629] [v2:127.0.0.1:6820/709392629,v1:127.0.0.1:6821/709392629] exists,up f5e9bb9e-535e-42dd-9e28-654a28c9dc8d 2024-05-08T17:04:25.421 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-08T17:04:25.421 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-08T17:04:25.421 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-08T17:04:25.421 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-05-08T17:04:25.421 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-05-08T17:04:25.421 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-05-08T17:04:25.421 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-05-08T17:04:25.422 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-05-08T17:04:25.422 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-05-08T17:04:25.422 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-05-08T17:04:25.422 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-05-08T17:04:25.422 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-05-08T17:04:25.422 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-05-08T17:04:25.523 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-05-08T17:04:25.523 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-05-08T17:04:25.523 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-05-08T17:04:25.523 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-05-08T17:04:25.523 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-05-08T17:04:25.523 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-05-08T17:04:26.103 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-05-08T17:04:26.103 INFO:tasks.workunit.client.0.smithi172.stderr:1 2024-05-08T17:04:26.103 INFO:tasks.workunit.client.0.smithi172.stderr:2 2024-05-08T17:04:26.103 INFO:tasks.workunit.client.0.smithi172.stderr:3' 2024-05-08T17:04:26.103 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-05-08T17:04:26.103 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T17:04:26.103 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-05-08T17:04:26.259 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=25769803818 2024-05-08T17:04:26.259 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 25769803818 2024-05-08T17:04:26.259 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803818' 2024-05-08T17:04:26.259 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T17:04:26.259 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-05-08T17:04:26.421 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=184683593736 2024-05-08T17:04:26.422 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 184683593736 2024-05-08T17:04:26.422 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803818 1-184683593736' 2024-05-08T17:04:26.422 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T17:04:26.422 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-05-08T17:04:26.570 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=227633266691 2024-05-08T17:04:26.571 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 227633266691 2024-05-08T17:04:26.571 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803818 1-184683593736 2-227633266691' 2024-05-08T17:04:26.571 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T17:04:26.571 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-05-08T17:04:26.722 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=120259084305 2024-05-08T17:04:26.722 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 120259084305 2024-05-08T17:04:26.722 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803818 1-184683593736 2-227633266691 3-120259084305' 2024-05-08T17:04:26.722 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-08T17:04:26.723 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-08T17:04:26.723 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-25769803818 2024-05-08T17:04:26.724 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-05-08T17:04:26.725 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-25769803818 2024-05-08T17:04:26.725 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-08T17:04:26.725 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=25769803818 2024-05-08T17:04:26.726 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 25769803818' 2024-05-08T17:04:26.726 INFO:tasks.workunit.client.0.smithi172.stderr:waiting osd.0 seq 25769803818 2024-05-08T17:04:26.726 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-05-08T17:04:27.132 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 25769803816 -lt 25769803818 2024-05-08T17:04:27.133 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-05-08T17:04:28.134 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-05-08T17:04:28.134 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-05-08T17:04:28.559 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 25769803819 -lt 25769803818 2024-05-08T17:04:28.559 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-08T17:04:28.560 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-184683593736 2024-05-08T17:04:28.560 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-08T17:04:28.561 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-05-08T17:04:28.562 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-184683593736 2024-05-08T17:04:28.562 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-08T17:04:28.564 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=184683593736 2024-05-08T17:04:28.564 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 184683593736' 2024-05-08T17:04:28.564 INFO:tasks.workunit.client.0.smithi172.stderr:waiting osd.1 seq 184683593736 2024-05-08T17:04:28.564 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-05-08T17:04:28.996 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 184683593737 -lt 184683593736 2024-05-08T17:04:28.996 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-08T17:04:28.996 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-227633266691 2024-05-08T17:04:28.997 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-08T17:04:28.998 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-05-08T17:04:28.999 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-227633266691 2024-05-08T17:04:28.999 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-08T17:04:29.000 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=227633266691 2024-05-08T17:04:29.000 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 227633266691' 2024-05-08T17:04:29.000 INFO:tasks.workunit.client.0.smithi172.stderr:waiting osd.2 seq 227633266691 2024-05-08T17:04:29.001 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-05-08T17:04:29.414 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 227633266691 -lt 227633266691 2024-05-08T17:04:29.414 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-08T17:04:29.415 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-120259084305 2024-05-08T17:04:29.415 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-08T17:04:29.416 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-05-08T17:04:29.418 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-120259084305 2024-05-08T17:04:29.418 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-08T17:04:29.419 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=120259084305 2024-05-08T17:04:29.419 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 120259084305' 2024-05-08T17:04:29.419 INFO:tasks.workunit.client.0.smithi172.stderr:waiting osd.3 seq 120259084305 2024-05-08T17:04:29.419 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-05-08T17:04:29.811 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 120259084305 -lt 120259084305 2024-05-08T17:04:29.811 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-05-08T17:04:29.812 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-05-08T17:04:29.812 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-05-08T17:04:30.321 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-05-08T17:04:30.321 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-05-08T17:04:30.322 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-05-08T17:04:30.322 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-05-08T17:04:30.322 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-05-08T17:04:30.322 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-05-08T17:04:30.322 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-05-08T17:04:30.322 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-05-08T17:04:30.731 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-05-08T17:04:30.732 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-05-08T17:04:30.732 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-05-08T17:04:30.732 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-05-08T17:04:31.273 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-05-08T17:04:31.273 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-05-08T17:04:31.273 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-05-08T17:04:31.273 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:217: set_size: rm -f td/test-erasure-eio/CORRUPT 2024-05-08T17:04:31.274 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:218: set_size: ceph osd unset noout 2024-05-08T17:04:31.947 INFO:tasks.workunit.client.0.smithi172.stderr:noout is unset 2024-05-08T17:04:31.965 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:243: rados_get_data_bad_size: rados_get td/test-erasure-eio pool-jerasure obj-size-81110-1-10 fail 2024-05-08T17:04:31.965 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2024-05-08T17:04:31.965 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2024-05-08T17:04:31.965 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=obj-size-81110-1-10 2024-05-08T17:04:31.965 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=fail 2024-05-08T17:04:31.965 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' fail = fail ']' 2024-05-08T17:04:31.966 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:114: rados_get: rados --pool pool-jerasure get obj-size-81110-1-10 td/test-erasure-eio/COPY 2024-05-08T17:04:32.125 INFO:tasks.workunit.client.0.smithi172.stderr:error getting pool-jerasure/obj-size-81110-1-10: (5) Input/output error 2024-05-08T17:04:32.128 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:115: rados_get: return 2024-05-08T17:04:32.129 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:244: rados_get_data_bad_size: rm td/test-erasure-eio/ORIGINAL 2024-05-08T17:04:32.132 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:324: TEST_rados_get_bad_size_shard_1: rados_get_data_bad_size td/test-erasure-eio 1 0 2024-05-08T17:04:32.132 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:222: rados_get_data_bad_size: local dir=td/test-erasure-eio 2024-05-08T17:04:32.132 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:223: rados_get_data_bad_size: shift 2024-05-08T17:04:32.132 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:224: rados_get_data_bad_size: local shard_id=1 2024-05-08T17:04:32.132 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:225: rados_get_data_bad_size: shift 2024-05-08T17:04:32.132 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:226: rados_get_data_bad_size: local bytes=0 2024-05-08T17:04:32.132 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:227: rados_get_data_bad_size: shift 2024-05-08T17:04:32.132 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:228: rados_get_data_bad_size: local mode=set 2024-05-08T17:04:32.132 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:230: rados_get_data_bad_size: local poolname=pool-jerasure 2024-05-08T17:04:32.132 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:231: rados_get_data_bad_size: local objname=obj-size-81110-1-0 2024-05-08T17:04:32.133 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:232: rados_get_data_bad_size: rados_put td/test-erasure-eio pool-jerasure obj-size-81110-1-0 2024-05-08T17:04:32.133 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:90: rados_put: local dir=td/test-erasure-eio 2024-05-08T17:04:32.133 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:91: rados_put: local poolname=pool-jerasure 2024-05-08T17:04:32.133 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:92: rados_put: local objname=obj-size-81110-1-0 2024-05-08T17:04:32.133 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-05-08T17:04:32.133 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 AAA 2024-05-08T17:04:32.133 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-05-08T17:04:32.133 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 BBB 2024-05-08T17:04:32.133 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-05-08T17:04:32.133 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 CCCC 2024-05-08T17:04:32.133 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-05-08T17:04:32.133 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 DDDD 2024-05-08T17:04:32.133 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:100: rados_put: rados --pool pool-jerasure put obj-size-81110-1-0 td/test-erasure-eio/ORIGINAL 2024-05-08T17:04:32.338 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:236: rados_get_data_bad_size: set_size obj-size-81110-1-0 td/test-erasure-eio 1 0 set 2024-05-08T17:04:32.338 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:192: set_size: local objname=obj-size-81110-1-0 2024-05-08T17:04:32.338 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:193: set_size: shift 2024-05-08T17:04:32.338 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:194: set_size: local dir=td/test-erasure-eio 2024-05-08T17:04:32.338 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:195: set_size: shift 2024-05-08T17:04:32.338 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:196: set_size: local shard_id=1 2024-05-08T17:04:32.338 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:197: set_size: shift 2024-05-08T17:04:32.338 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:198: set_size: local bytes=0 2024-05-08T17:04:32.339 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:199: set_size: shift 2024-05-08T17:04:32.339 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:200: set_size: local mode=set 2024-05-08T17:04:32.339 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:202: set_size: local poolname=pool-jerasure 2024-05-08T17:04:32.339 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: get_osds pool-jerasure obj-size-81110-1-0 2024-05-08T17:04:32.339 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2024-05-08T17:04:32.339 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-size-81110-1-0 2024-05-08T17:04:32.339 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure obj-size-81110-1-0 2024-05-08T17:04:32.339 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-05-08T17:04:32.751 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-05-08T17:04:32.752 INFO:tasks.workunit.client.0.smithi172.stderr:1 2024-05-08T17:04:32.752 INFO:tasks.workunit.client.0.smithi172.stderr:2' 2024-05-08T17:04:32.752 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 2 2024-05-08T17:04:32.752 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: initial_osds=('3' '1' '2') 2024-05-08T17:04:32.752 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: local -a initial_osds 2024-05-08T17:04:32.752 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:204: set_size: local osd_id=1 2024-05-08T17:04:32.752 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:205: set_size: ceph osd set noout 2024-05-08T17:04:33.537 INFO:tasks.workunit.client.0.smithi172.stderr:noout is set 2024-05-08T17:04:33.559 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:206: set_size: '[' set = add ']' 2024-05-08T17:04:33.559 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:210: set_size: '[' 0 = 0 ']' 2024-05-08T17:04:33.559 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:212: set_size: touch td/test-erasure-eio/CORRUPT 2024-05-08T17:04:33.561 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:216: set_size: objectstore_tool td/test-erasure-eio 1 obj-size-81110-1-0 set-bytes td/test-erasure-eio/CORRUPT 2024-05-08T17:04:33.562 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/test-erasure-eio 2024-05-08T17:04:33.562 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-05-08T17:04:33.562 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-05-08T17:04:33.562 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-05-08T17:04:33.562 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/test-erasure-eio 1 obj-size-81110-1-0 set-bytes td/test-erasure-eio/CORRUPT 2024-05-08T17:04:33.562 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/test-erasure-eio 2024-05-08T17:04:33.562 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-05-08T17:04:33.562 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-05-08T17:04:33.562 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-05-08T17:04:33.562 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/test-erasure-eio TERM osd.1 2024-05-08T17:04:33.562 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-08T17:04:33.562 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-08T17:04:33.563 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-08T17:04:33.563 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-08T17:04:33.563 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-08T17:04:34.874 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-08T17:04:34.874 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/test-erasure-eio 1 obj-size-81110-1-0 set-bytes td/test-erasure-eio/CORRUPT 2024-05-08T17:04:34.874 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/test-erasure-eio 2024-05-08T17:04:34.874 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-05-08T17:04:34.874 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-05-08T17:04:34.874 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-05-08T17:04:34.874 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/test-erasure-eio/1 2024-05-08T17:04:34.874 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/test-erasure-eio/1 obj-size-81110-1-0 set-bytes td/test-erasure-eio/CORRUPT 2024-05-08T17:04:36.762 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/test-erasure-eio 1 2024-05-08T17:04:36.763 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/test-erasure-eio 2024-05-08T17:04:36.763 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-05-08T17:04:36.763 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-05-08T17:04:36.763 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-05-08T17:04:36.763 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/test-erasure-eio/1 2024-05-08T17:04:36.763 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=4457b0f1-028b-482b-8ee3-7ee05861e6cc --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-05-08T17:04:36.763 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-08T17:04:36.763 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-05-08T17:04:36.763 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-08T17:04:36.763 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/1' 2024-05-08T17:04:36.763 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/1/journal' 2024-05-08T17:04:36.764 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-05-08T17:04:36.764 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-05-08T17:04:36.764 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-05-08T17:04:36.764 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-05-08T17:04:36.764 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-08T17:04:36.764 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-08T17:04:36.765 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-08T17:04:36.765 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T17:04:36.765 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81110 2024-05-08T17:04:36.765 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.81110/$cluster-$name.asok' 2024-05-08T17:04:36.765 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.81110/$cluster-$name.asok' 2024-05-08T17:04:36.765 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-05-08T17:04:36.765 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-05-08T17:04:36.765 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-05-08T17:04:36.766 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-08T17:04:36.766 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-08T17:04:36.766 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-08T17:04:36.766 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-08T17:04:36.766 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-05-08T17:04:36.766 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-05-08T17:04:36.766 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/test-erasure-eio/1 2024-05-08T17:04:36.766 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-05-08T17:04:36.766 INFO:tasks.workunit.client.0.smithi172.stderr:start osd.1 2024-05-08T17:04:36.767 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=4457b0f1-028b-482b-8ee3-7ee05861e6cc --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81110/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-05-08T17:04:36.767 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/test-erasure-eio/1/whoami 2024-05-08T17:04:36.768 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-05-08T17:04:36.769 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-05-08T17:04:36.769 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-05-08T17:04:36.769 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-05-08T17:04:36.800 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:04:36.806+0000 7f72d133a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T17:04:36.803 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:04:36.810+0000 7f72d133a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T17:04:36.804 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:04:36.810+0000 7f72d133a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T17:04:37.181 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-05-08T17:04:37.181 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-08T17:04:37.181 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-05-08T17:04:37.181 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-08T17:04:37.181 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-08T17:04:37.181 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:04:37.182 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-08T17:04:37.182 INFO:tasks.workunit.client.0.smithi172.stderr:0 2024-05-08T17:04:37.182 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T17:04:37.187 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:04:37.352 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:04:37.358+0000 7f72d133a7c0 -1 Falling back to public interface 2024-05-08T17:04:37.664 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:04:38.666 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:04:38.666 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:04:38.666 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-08T17:04:38.666 INFO:tasks.workunit.client.0.smithi172.stderr:1 2024-05-08T17:04:38.667 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:04:38.667 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T17:04:39.069 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:04:40.047 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:04:40.054+0000 7f72d133a7c0 -1 osd.1 57 log_to_monitors true 2024-05-08T17:04:40.070 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:04:40.070 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:04:40.070 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-08T17:04:40.070 INFO:tasks.workunit.client.0.smithi172.stderr:2 2024-05-08T17:04:40.071 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T17:04:40.071 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:04:40.740 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:04:41.742 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:04:41.742 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:04:41.742 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-05-08T17:04:41.742 INFO:tasks.workunit.client.0.smithi172.stderr:3 2024-05-08T17:04:41.742 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:04:41.743 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T17:04:42.496 INFO:tasks.workunit.client.0.smithi172.stderr:osd.1 up in weight 1 up_from 62 up_thru 62 down_at 59 last_clean_interval [43,57) [v2:127.0.0.1:6810/1647412275,v1:127.0.0.1:6811/1647412275] [v2:127.0.0.1:6812/1647412275,v1:127.0.0.1:6813/1647412275] exists,up a0a953bf-e981-4e54-97d5-d7c2101753a9 2024-05-08T17:04:42.496 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-08T17:04:42.496 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-08T17:04:42.496 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-08T17:04:42.496 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-05-08T17:04:42.496 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-05-08T17:04:42.496 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-05-08T17:04:42.496 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-05-08T17:04:42.497 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-05-08T17:04:42.497 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-05-08T17:04:42.497 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-05-08T17:04:42.498 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-05-08T17:04:42.498 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-05-08T17:04:42.498 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-05-08T17:04:42.617 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-05-08T17:04:42.618 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-05-08T17:04:42.618 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-05-08T17:04:42.618 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-05-08T17:04:42.618 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-05-08T17:04:42.618 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-05-08T17:04:43.085 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-05-08T17:04:43.085 INFO:tasks.workunit.client.0.smithi172.stderr:1 2024-05-08T17:04:43.085 INFO:tasks.workunit.client.0.smithi172.stderr:2 2024-05-08T17:04:43.085 INFO:tasks.workunit.client.0.smithi172.stderr:3' 2024-05-08T17:04:43.085 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-05-08T17:04:43.085 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T17:04:43.085 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-05-08T17:04:43.269 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=25769803824 2024-05-08T17:04:43.269 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 25769803824 2024-05-08T17:04:43.269 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803824' 2024-05-08T17:04:43.269 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T17:04:43.269 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-05-08T17:04:43.424 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=266287972355 2024-05-08T17:04:43.424 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 266287972355 2024-05-08T17:04:43.424 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803824 1-266287972355' 2024-05-08T17:04:43.424 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T17:04:43.424 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-05-08T17:04:43.575 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=227633266696 2024-05-08T17:04:43.575 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 227633266696 2024-05-08T17:04:43.575 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803824 1-266287972355 2-227633266696' 2024-05-08T17:04:43.575 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T17:04:43.576 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-05-08T17:04:43.730 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=120259084310 2024-05-08T17:04:43.731 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 120259084310 2024-05-08T17:04:43.731 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803824 1-266287972355 2-227633266696 3-120259084310' 2024-05-08T17:04:43.731 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-08T17:04:43.732 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-25769803824 2024-05-08T17:04:43.732 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-08T17:04:43.732 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-05-08T17:04:43.733 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-25769803824 2024-05-08T17:04:43.733 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-08T17:04:43.734 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=25769803824 2024-05-08T17:04:43.734 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 25769803824' 2024-05-08T17:04:43.734 INFO:tasks.workunit.client.0.smithi172.stderr:waiting osd.0 seq 25769803824 2024-05-08T17:04:43.734 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-05-08T17:04:44.139 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 25769803824 -lt 25769803824 2024-05-08T17:04:44.139 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-08T17:04:44.140 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-266287972355 2024-05-08T17:04:44.140 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-08T17:04:44.142 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-05-08T17:04:44.142 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-266287972355 2024-05-08T17:04:44.143 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-08T17:04:44.144 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=266287972355 2024-05-08T17:04:44.144 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 266287972355' 2024-05-08T17:04:44.144 INFO:tasks.workunit.client.0.smithi172.stderr:waiting osd.1 seq 266287972355 2024-05-08T17:04:44.144 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-05-08T17:04:44.592 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 1 -lt 266287972355 2024-05-08T17:04:44.592 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-05-08T17:04:45.594 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-05-08T17:04:45.594 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-05-08T17:04:46.042 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 266287972355 -lt 266287972355 2024-05-08T17:04:46.042 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-08T17:04:46.043 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-227633266696 2024-05-08T17:04:46.044 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-08T17:04:46.046 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-05-08T17:04:46.047 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-227633266696 2024-05-08T17:04:46.047 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-08T17:04:46.048 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=227633266696 2024-05-08T17:04:46.048 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 227633266696' 2024-05-08T17:04:46.048 INFO:tasks.workunit.client.0.smithi172.stderr:waiting osd.2 seq 227633266696 2024-05-08T17:04:46.048 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-05-08T17:04:46.462 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 227633266697 -lt 227633266696 2024-05-08T17:04:46.462 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-08T17:04:46.463 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-08T17:04:46.463 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-120259084310 2024-05-08T17:04:46.464 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-05-08T17:04:46.465 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-120259084310 2024-05-08T17:04:46.465 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-08T17:04:46.467 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=120259084310 2024-05-08T17:04:46.467 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 120259084310' 2024-05-08T17:04:46.467 INFO:tasks.workunit.client.0.smithi172.stderr:waiting osd.3 seq 120259084310 2024-05-08T17:04:46.467 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-05-08T17:04:46.857 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 120259084310 -lt 120259084310 2024-05-08T17:04:46.858 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-05-08T17:04:46.859 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-05-08T17:04:46.859 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-05-08T17:04:47.402 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-05-08T17:04:47.402 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-05-08T17:04:47.403 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-05-08T17:04:47.403 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-05-08T17:04:47.403 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-05-08T17:04:47.403 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-05-08T17:04:47.403 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-05-08T17:04:47.403 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-05-08T17:04:47.787 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-05-08T17:04:47.787 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-05-08T17:04:47.788 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-05-08T17:04:47.788 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-05-08T17:04:48.353 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-05-08T17:04:48.353 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-05-08T17:04:48.353 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-05-08T17:04:48.353 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:217: set_size: rm -f td/test-erasure-eio/CORRUPT 2024-05-08T17:04:48.354 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:218: set_size: ceph osd unset noout 2024-05-08T17:04:49.069 INFO:tasks.workunit.client.0.smithi172.stderr:noout is unset 2024-05-08T17:04:49.091 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:238: rados_get_data_bad_size: rados_get td/test-erasure-eio pool-jerasure obj-size-81110-1-0 2024-05-08T17:04:49.091 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2024-05-08T17:04:49.091 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2024-05-08T17:04:49.091 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=obj-size-81110-1-0 2024-05-08T17:04:49.091 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=ok 2024-05-08T17:04:49.091 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' ok = fail ']' 2024-05-08T17:04:49.091 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:120: rados_get: rados --pool pool-jerasure get obj-size-81110-1-0 td/test-erasure-eio/COPY 2024-05-08T17:04:49.296 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:121: rados_get: diff td/test-erasure-eio/ORIGINAL td/test-erasure-eio/COPY 2024-05-08T17:04:49.297 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:122: rados_get: rm td/test-erasure-eio/COPY 2024-05-08T17:04:49.312 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:241: rados_get_data_bad_size: expr 1 + 1 2024-05-08T17:04:49.315 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:241: rados_get_data_bad_size: shard_id=2 2024-05-08T17:04:49.315 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:242: rados_get_data_bad_size: set_size obj-size-81110-1-0 td/test-erasure-eio 2 0 set 2024-05-08T17:04:49.316 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:192: set_size: local objname=obj-size-81110-1-0 2024-05-08T17:04:49.316 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:193: set_size: shift 2024-05-08T17:04:49.316 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:194: set_size: local dir=td/test-erasure-eio 2024-05-08T17:04:49.316 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:195: set_size: shift 2024-05-08T17:04:49.316 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:196: set_size: local shard_id=2 2024-05-08T17:04:49.316 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:197: set_size: shift 2024-05-08T17:04:49.316 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:198: set_size: local bytes=0 2024-05-08T17:04:49.316 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:199: set_size: shift 2024-05-08T17:04:49.316 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:200: set_size: local mode=set 2024-05-08T17:04:49.317 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:202: set_size: local poolname=pool-jerasure 2024-05-08T17:04:49.317 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: get_osds pool-jerasure obj-size-81110-1-0 2024-05-08T17:04:49.317 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2024-05-08T17:04:49.317 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-size-81110-1-0 2024-05-08T17:04:49.318 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure obj-size-81110-1-0 2024-05-08T17:04:49.318 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-05-08T17:04:49.729 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-05-08T17:04:49.729 INFO:tasks.workunit.client.0.smithi172.stderr:1 2024-05-08T17:04:49.729 INFO:tasks.workunit.client.0.smithi172.stderr:2' 2024-05-08T17:04:49.729 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 2 2024-05-08T17:04:49.729 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: initial_osds=('3' '1' '2') 2024-05-08T17:04:49.729 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: local -a initial_osds 2024-05-08T17:04:49.729 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:204: set_size: local osd_id=2 2024-05-08T17:04:49.729 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:205: set_size: ceph osd set noout 2024-05-08T17:04:50.642 INFO:tasks.workunit.client.0.smithi172.stderr:noout is set 2024-05-08T17:04:50.665 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:206: set_size: '[' set = add ']' 2024-05-08T17:04:50.665 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:210: set_size: '[' 0 = 0 ']' 2024-05-08T17:04:50.665 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:212: set_size: touch td/test-erasure-eio/CORRUPT 2024-05-08T17:04:50.669 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:216: set_size: objectstore_tool td/test-erasure-eio 2 obj-size-81110-1-0 set-bytes td/test-erasure-eio/CORRUPT 2024-05-08T17:04:50.669 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/test-erasure-eio 2024-05-08T17:04:50.669 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-05-08T17:04:50.670 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=2 2024-05-08T17:04:50.670 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-05-08T17:04:50.670 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/test-erasure-eio 2 obj-size-81110-1-0 set-bytes td/test-erasure-eio/CORRUPT 2024-05-08T17:04:50.670 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/test-erasure-eio 2024-05-08T17:04:50.670 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-05-08T17:04:50.670 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=2 2024-05-08T17:04:50.671 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-05-08T17:04:50.671 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/test-erasure-eio TERM osd.2 2024-05-08T17:04:50.671 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-08T17:04:50.671 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-08T17:04:50.671 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-08T17:04:50.671 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-08T17:04:50.672 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-08T17:04:51.982 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-08T17:04:51.982 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/test-erasure-eio 2 obj-size-81110-1-0 set-bytes td/test-erasure-eio/CORRUPT 2024-05-08T17:04:51.982 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/test-erasure-eio 2024-05-08T17:04:51.982 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-05-08T17:04:51.982 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=2 2024-05-08T17:04:51.983 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-05-08T17:04:51.983 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/test-erasure-eio/2 2024-05-08T17:04:51.983 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/test-erasure-eio/2 obj-size-81110-1-0 set-bytes td/test-erasure-eio/CORRUPT 2024-05-08T17:04:53.807 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/test-erasure-eio 2 2024-05-08T17:04:53.807 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/test-erasure-eio 2024-05-08T17:04:53.807 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-05-08T17:04:53.807 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-05-08T17:04:53.807 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-05-08T17:04:53.807 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/test-erasure-eio/2 2024-05-08T17:04:53.807 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=4457b0f1-028b-482b-8ee3-7ee05861e6cc --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-05-08T17:04:53.807 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-08T17:04:53.807 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-05-08T17:04:53.807 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-08T17:04:53.807 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/2' 2024-05-08T17:04:53.807 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/2/journal' 2024-05-08T17:04:53.808 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-05-08T17:04:53.808 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-05-08T17:04:53.808 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-05-08T17:04:53.808 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-05-08T17:04:53.808 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-08T17:04:53.808 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-08T17:04:53.810 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-08T17:04:53.810 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T17:04:53.810 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81110 2024-05-08T17:04:53.810 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.81110/$cluster-$name.asok' 2024-05-08T17:04:53.810 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.81110/$cluster-$name.asok' 2024-05-08T17:04:53.811 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-05-08T17:04:53.811 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-05-08T17:04:53.811 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-05-08T17:04:53.811 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-08T17:04:53.811 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-08T17:04:53.811 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-08T17:04:53.811 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-08T17:04:53.811 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-05-08T17:04:53.811 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-05-08T17:04:53.811 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/test-erasure-eio/2 2024-05-08T17:04:53.812 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-05-08T17:04:53.812 INFO:tasks.workunit.client.0.smithi172.stderr:start osd.2 2024-05-08T17:04:53.812 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=4457b0f1-028b-482b-8ee3-7ee05861e6cc --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81110/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-05-08T17:04:53.812 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/test-erasure-eio/2/whoami 2024-05-08T17:04:53.812 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-05-08T17:04:53.813 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-05-08T17:04:53.813 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-05-08T17:04:53.813 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-05-08T17:04:53.845 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:04:53.850+0000 7f9af90cb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T17:04:53.863 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:04:53.870+0000 7f9af90cb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T17:04:53.864 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:04:53.870+0000 7f9af90cb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T17:04:54.240 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-05-08T17:04:54.240 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-08T17:04:54.241 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-05-08T17:04:54.241 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-08T17:04:54.241 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-08T17:04:54.241 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:04:54.241 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-08T17:04:54.241 INFO:tasks.workunit.client.0.smithi172.stderr:0 2024-05-08T17:04:54.241 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:04:54.241 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T17:04:54.416 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:04:54.422+0000 7f9af90cb7c0 -1 Falling back to public interface 2024-05-08T17:04:54.677 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:04:55.680 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:04:55.680 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:04:55.680 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-08T17:04:55.681 INFO:tasks.workunit.client.0.smithi172.stderr:1 2024-05-08T17:04:55.681 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:04:55.681 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T17:04:56.138 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:04:57.074 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:04:57.078+0000 7f9af90cb7c0 -1 osd.2 67 log_to_monitors true 2024-05-08T17:04:57.140 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:04:57.140 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:04:57.140 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-08T17:04:57.140 INFO:tasks.workunit.client.0.smithi172.stderr:2 2024-05-08T17:04:57.140 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:04:57.141 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T17:04:57.624 INFO:tasks.workunit.client.0.smithi172.stderr:osd.2 up in weight 1 up_from 73 up_thru 59 down_at 70 last_clean_interval [53,67) [v2:127.0.0.1:6818/3647848075,v1:127.0.0.1:6819/3647848075] [v2:127.0.0.1:6820/3647848075,v1:127.0.0.1:6821/3647848075] exists,up f5e9bb9e-535e-42dd-9e28-654a28c9dc8d 2024-05-08T17:04:57.624 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-08T17:04:57.624 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-08T17:04:57.625 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-08T17:04:57.625 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-05-08T17:04:57.625 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-05-08T17:04:57.625 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-05-08T17:04:57.625 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-05-08T17:04:57.626 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-05-08T17:04:57.626 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-05-08T17:04:57.626 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-05-08T17:04:57.626 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-05-08T17:04:57.626 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-05-08T17:04:57.627 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-05-08T17:04:57.728 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-05-08T17:04:57.729 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-05-08T17:04:57.729 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-05-08T17:04:57.729 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-05-08T17:04:57.729 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-05-08T17:04:57.729 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-05-08T17:04:58.387 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-05-08T17:04:58.387 INFO:tasks.workunit.client.0.smithi172.stderr:1 2024-05-08T17:04:58.387 INFO:tasks.workunit.client.0.smithi172.stderr:2 2024-05-08T17:04:58.387 INFO:tasks.workunit.client.0.smithi172.stderr:3' 2024-05-08T17:04:58.388 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-05-08T17:04:58.388 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T17:04:58.388 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-05-08T17:04:58.652 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=25769803829 2024-05-08T17:04:58.652 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 25769803829 2024-05-08T17:04:58.652 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803829' 2024-05-08T17:04:58.652 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T17:04:58.652 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-05-08T17:04:58.827 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=266287972360 2024-05-08T17:04:58.827 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 266287972360 2024-05-08T17:04:58.828 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803829 1-266287972360' 2024-05-08T17:04:58.828 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T17:04:58.828 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-05-08T17:04:58.977 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=313532612611 2024-05-08T17:04:58.978 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 313532612611 2024-05-08T17:04:58.978 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803829 1-266287972360 2-313532612611' 2024-05-08T17:04:58.978 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T17:04:58.978 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-05-08T17:04:59.190 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=120259084315 2024-05-08T17:04:59.191 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 120259084315 2024-05-08T17:04:59.191 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803829 1-266287972360 2-313532612611 3-120259084315' 2024-05-08T17:04:59.191 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-08T17:04:59.191 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-25769803829 2024-05-08T17:04:59.191 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-08T17:04:59.193 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-05-08T17:04:59.194 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-25769803829 2024-05-08T17:04:59.194 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-08T17:04:59.195 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=25769803829 2024-05-08T17:04:59.195 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 25769803829' 2024-05-08T17:04:59.195 INFO:tasks.workunit.client.0.smithi172.stderr:waiting osd.0 seq 25769803829 2024-05-08T17:04:59.196 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-05-08T17:04:59.627 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 25769803829 -lt 25769803829 2024-05-08T17:04:59.627 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-08T17:04:59.628 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-266287972360 2024-05-08T17:04:59.628 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-08T17:04:59.629 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-05-08T17:04:59.630 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-266287972360 2024-05-08T17:04:59.630 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-08T17:04:59.632 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=266287972360 2024-05-08T17:04:59.632 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 266287972360' 2024-05-08T17:04:59.632 INFO:tasks.workunit.client.0.smithi172.stderr:waiting osd.1 seq 266287972360 2024-05-08T17:04:59.632 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-05-08T17:05:00.091 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 266287972360 -lt 266287972360 2024-05-08T17:05:00.091 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-08T17:05:00.091 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-313532612611 2024-05-08T17:05:00.092 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-08T17:05:00.093 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-05-08T17:05:00.093 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-313532612611 2024-05-08T17:05:00.094 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-08T17:05:00.094 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=313532612611 2024-05-08T17:05:00.094 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 313532612611' 2024-05-08T17:05:00.095 INFO:tasks.workunit.client.0.smithi172.stderr:waiting osd.2 seq 313532612611 2024-05-08T17:05:00.095 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-05-08T17:05:00.556 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 313532612611 -lt 313532612611 2024-05-08T17:05:00.556 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-08T17:05:00.557 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-120259084315 2024-05-08T17:05:00.557 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-08T17:05:00.559 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-05-08T17:05:00.559 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-120259084315 2024-05-08T17:05:00.560 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-08T17:05:00.561 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=120259084315 2024-05-08T17:05:00.561 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 120259084315' 2024-05-08T17:05:00.561 INFO:tasks.workunit.client.0.smithi172.stderr:waiting osd.3 seq 120259084315 2024-05-08T17:05:00.562 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-05-08T17:05:01.012 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 120259084315 -lt 120259084315 2024-05-08T17:05:01.013 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-05-08T17:05:01.013 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-05-08T17:05:01.014 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-05-08T17:05:01.566 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-05-08T17:05:01.566 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-05-08T17:05:01.567 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-05-08T17:05:01.567 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-05-08T17:05:01.567 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-05-08T17:05:01.567 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-05-08T17:05:01.567 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-05-08T17:05:01.567 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-05-08T17:05:01.966 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-05-08T17:05:01.966 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-05-08T17:05:01.967 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-05-08T17:05:01.967 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-05-08T17:05:02.502 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-05-08T17:05:02.502 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-05-08T17:05:02.502 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-05-08T17:05:02.502 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:217: set_size: rm -f td/test-erasure-eio/CORRUPT 2024-05-08T17:05:02.503 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:218: set_size: ceph osd unset noout 2024-05-08T17:05:03.267 INFO:tasks.workunit.client.0.smithi172.stderr:noout is unset 2024-05-08T17:05:03.288 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:243: rados_get_data_bad_size: rados_get td/test-erasure-eio pool-jerasure obj-size-81110-1-0 fail 2024-05-08T17:05:03.288 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2024-05-08T17:05:03.288 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2024-05-08T17:05:03.288 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=obj-size-81110-1-0 2024-05-08T17:05:03.288 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=fail 2024-05-08T17:05:03.288 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' fail = fail ']' 2024-05-08T17:05:03.288 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:114: rados_get: rados --pool pool-jerasure get obj-size-81110-1-0 td/test-erasure-eio/COPY 2024-05-08T17:05:03.428 INFO:tasks.workunit.client.0.smithi172.stderr:error getting pool-jerasure/obj-size-81110-1-0: (5) Input/output error 2024-05-08T17:05:03.432 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:115: rados_get: return 2024-05-08T17:05:03.432 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:244: rados_get_data_bad_size: rm td/test-erasure-eio/ORIGINAL 2024-05-08T17:05:03.436 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:325: TEST_rados_get_bad_size_shard_1: rados_get_data_bad_size td/test-erasure-eio 1 256 add 2024-05-08T17:05:03.436 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:222: rados_get_data_bad_size: local dir=td/test-erasure-eio 2024-05-08T17:05:03.436 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:223: rados_get_data_bad_size: shift 2024-05-08T17:05:03.436 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:224: rados_get_data_bad_size: local shard_id=1 2024-05-08T17:05:03.436 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:225: rados_get_data_bad_size: shift 2024-05-08T17:05:03.436 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:226: rados_get_data_bad_size: local bytes=256 2024-05-08T17:05:03.436 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:227: rados_get_data_bad_size: shift 2024-05-08T17:05:03.436 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:228: rados_get_data_bad_size: local mode=add 2024-05-08T17:05:03.436 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:230: rados_get_data_bad_size: local poolname=pool-jerasure 2024-05-08T17:05:03.436 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:231: rados_get_data_bad_size: local objname=obj-size-81110-1-256 2024-05-08T17:05:03.436 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:232: rados_get_data_bad_size: rados_put td/test-erasure-eio pool-jerasure obj-size-81110-1-256 2024-05-08T17:05:03.437 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:90: rados_put: local dir=td/test-erasure-eio 2024-05-08T17:05:03.437 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:91: rados_put: local poolname=pool-jerasure 2024-05-08T17:05:03.437 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:92: rados_put: local objname=obj-size-81110-1-256 2024-05-08T17:05:03.437 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-05-08T17:05:03.437 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 AAA 2024-05-08T17:05:03.437 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-05-08T17:05:03.437 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 BBB 2024-05-08T17:05:03.437 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-05-08T17:05:03.437 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 CCCC 2024-05-08T17:05:03.437 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-05-08T17:05:03.437 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 DDDD 2024-05-08T17:05:03.437 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:100: rados_put: rados --pool pool-jerasure put obj-size-81110-1-256 td/test-erasure-eio/ORIGINAL 2024-05-08T17:05:03.709 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:236: rados_get_data_bad_size: set_size obj-size-81110-1-256 td/test-erasure-eio 1 256 add 2024-05-08T17:05:03.710 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:192: set_size: local objname=obj-size-81110-1-256 2024-05-08T17:05:03.710 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:193: set_size: shift 2024-05-08T17:05:03.710 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:194: set_size: local dir=td/test-erasure-eio 2024-05-08T17:05:03.710 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:195: set_size: shift 2024-05-08T17:05:03.710 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:196: set_size: local shard_id=1 2024-05-08T17:05:03.710 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:197: set_size: shift 2024-05-08T17:05:03.710 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:198: set_size: local bytes=256 2024-05-08T17:05:03.710 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:199: set_size: shift 2024-05-08T17:05:03.710 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:200: set_size: local mode=add 2024-05-08T17:05:03.710 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:202: set_size: local poolname=pool-jerasure 2024-05-08T17:05:03.711 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: get_osds pool-jerasure obj-size-81110-1-256 2024-05-08T17:05:03.711 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2024-05-08T17:05:03.711 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-size-81110-1-256 2024-05-08T17:05:03.712 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure obj-size-81110-1-256 2024-05-08T17:05:03.712 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-05-08T17:05:04.164 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-05-08T17:05:04.164 INFO:tasks.workunit.client.0.smithi172.stderr:1 2024-05-08T17:05:04.164 INFO:tasks.workunit.client.0.smithi172.stderr:2' 2024-05-08T17:05:04.164 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 2 2024-05-08T17:05:04.164 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: initial_osds=('3' '1' '2') 2024-05-08T17:05:04.164 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: local -a initial_osds 2024-05-08T17:05:04.164 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:204: set_size: local osd_id=1 2024-05-08T17:05:04.164 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:205: set_size: ceph osd set noout 2024-05-08T17:05:04.816 INFO:tasks.workunit.client.0.smithi172.stderr:noout is set 2024-05-08T17:05:04.838 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:206: set_size: '[' add = add ']' 2024-05-08T17:05:04.839 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:208: set_size: objectstore_tool td/test-erasure-eio 1 obj-size-81110-1-256 get-bytes td/test-erasure-eio/CORRUPT 2024-05-08T17:05:04.839 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/test-erasure-eio 2024-05-08T17:05:04.839 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-05-08T17:05:04.839 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-05-08T17:05:04.839 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-05-08T17:05:04.839 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/test-erasure-eio 1 obj-size-81110-1-256 get-bytes td/test-erasure-eio/CORRUPT 2024-05-08T17:05:04.840 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/test-erasure-eio 2024-05-08T17:05:04.840 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-05-08T17:05:04.840 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-05-08T17:05:04.840 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-05-08T17:05:04.840 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/test-erasure-eio TERM osd.1 2024-05-08T17:05:04.840 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-08T17:05:04.840 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-08T17:05:04.841 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-08T17:05:04.841 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-08T17:05:04.841 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-08T17:05:06.154 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-08T17:05:06.154 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/test-erasure-eio 1 obj-size-81110-1-256 get-bytes td/test-erasure-eio/CORRUPT 2024-05-08T17:05:06.154 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/test-erasure-eio 2024-05-08T17:05:06.154 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-05-08T17:05:06.154 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-05-08T17:05:06.154 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-05-08T17:05:06.154 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/test-erasure-eio/1 2024-05-08T17:05:06.154 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/test-erasure-eio/1 obj-size-81110-1-256 get-bytes td/test-erasure-eio/CORRUPT 2024-05-08T17:05:08.050 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/test-erasure-eio 1 2024-05-08T17:05:08.050 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/test-erasure-eio 2024-05-08T17:05:08.050 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-05-08T17:05:08.050 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-05-08T17:05:08.050 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-05-08T17:05:08.050 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/test-erasure-eio/1 2024-05-08T17:05:08.051 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=4457b0f1-028b-482b-8ee3-7ee05861e6cc --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-05-08T17:05:08.051 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-08T17:05:08.051 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-05-08T17:05:08.051 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-08T17:05:08.051 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/1' 2024-05-08T17:05:08.051 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/1/journal' 2024-05-08T17:05:08.051 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-05-08T17:05:08.051 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-05-08T17:05:08.051 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-05-08T17:05:08.051 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-05-08T17:05:08.051 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-08T17:05:08.051 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-08T17:05:08.051 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-08T17:05:08.051 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T17:05:08.052 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81110 2024-05-08T17:05:08.053 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.81110/$cluster-$name.asok' 2024-05-08T17:05:08.053 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.81110/$cluster-$name.asok' 2024-05-08T17:05:08.053 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-05-08T17:05:08.053 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-05-08T17:05:08.053 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-05-08T17:05:08.053 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-08T17:05:08.053 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-08T17:05:08.053 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-08T17:05:08.053 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-08T17:05:08.054 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-05-08T17:05:08.054 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-05-08T17:05:08.054 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/test-erasure-eio/1 2024-05-08T17:05:08.054 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-05-08T17:05:08.054 INFO:tasks.workunit.client.0.smithi172.stderr:start osd.1 2024-05-08T17:05:08.054 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=4457b0f1-028b-482b-8ee3-7ee05861e6cc --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81110/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-05-08T17:05:08.054 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/test-erasure-eio/1/whoami 2024-05-08T17:05:08.055 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-05-08T17:05:08.056 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-05-08T17:05:08.056 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-05-08T17:05:08.056 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-05-08T17:05:08.087 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:05:08.094+0000 7fb85be2b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T17:05:08.090 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:05:08.094+0000 7fb85be2b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T17:05:08.092 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:05:08.098+0000 7fb85be2b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T17:05:08.467 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-05-08T17:05:08.467 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-08T17:05:08.467 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-05-08T17:05:08.467 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-08T17:05:08.467 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-08T17:05:08.467 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:05:08.467 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-08T17:05:08.467 INFO:tasks.workunit.client.0.smithi172.stderr:0 2024-05-08T17:05:08.467 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T17:05:08.467 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:05:08.632 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:05:08.638+0000 7fb85be2b7c0 -1 Falling back to public interface 2024-05-08T17:05:08.911 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:05:09.913 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:05:09.913 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:05:09.913 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-08T17:05:09.913 INFO:tasks.workunit.client.0.smithi172.stderr:1 2024-05-08T17:05:09.914 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:05:09.914 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T17:05:10.322 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:05:11.306 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:05:11.310+0000 7fb85be2b7c0 -1 osd.1 79 log_to_monitors true 2024-05-08T17:05:11.324 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:05:11.324 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:05:11.324 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-08T17:05:11.324 INFO:tasks.workunit.client.0.smithi172.stderr:2 2024-05-08T17:05:11.324 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T17:05:11.324 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:05:11.855 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:05:12.857 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:05:12.857 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:05:12.857 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-05-08T17:05:12.857 INFO:tasks.workunit.client.0.smithi172.stderr:3 2024-05-08T17:05:12.858 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:05:12.858 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T17:05:13.572 INFO:tasks.workunit.client.0.smithi172.stderr:osd.1 up in weight 1 up_from 84 up_thru 84 down_at 80 last_clean_interval [62,79) [v2:127.0.0.1:6810/3053010802,v1:127.0.0.1:6811/3053010802] [v2:127.0.0.1:6812/3053010802,v1:127.0.0.1:6813/3053010802] exists,up a0a953bf-e981-4e54-97d5-d7c2101753a9 2024-05-08T17:05:13.572 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-08T17:05:13.572 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-08T17:05:13.572 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-08T17:05:13.572 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-05-08T17:05:13.572 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-05-08T17:05:13.572 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-05-08T17:05:13.572 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-05-08T17:05:13.573 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-05-08T17:05:13.574 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-05-08T17:05:13.574 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-05-08T17:05:13.575 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-05-08T17:05:13.575 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-05-08T17:05:13.575 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-05-08T17:05:13.686 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-05-08T17:05:13.687 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-05-08T17:05:13.687 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-05-08T17:05:13.687 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-05-08T17:05:13.687 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-05-08T17:05:13.687 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-05-08T17:05:14.125 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-05-08T17:05:14.125 INFO:tasks.workunit.client.0.smithi172.stderr:1 2024-05-08T17:05:14.125 INFO:tasks.workunit.client.0.smithi172.stderr:2 2024-05-08T17:05:14.125 INFO:tasks.workunit.client.0.smithi172.stderr:3' 2024-05-08T17:05:14.126 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-05-08T17:05:14.126 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T17:05:14.126 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-05-08T17:05:14.295 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=25769803834 2024-05-08T17:05:14.295 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 25769803834 2024-05-08T17:05:14.296 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803834' 2024-05-08T17:05:14.296 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T17:05:14.296 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-05-08T17:05:14.445 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=360777252867 2024-05-08T17:05:14.445 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 360777252867 2024-05-08T17:05:14.445 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803834 1-360777252867' 2024-05-08T17:05:14.445 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T17:05:14.446 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-05-08T17:05:14.596 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=313532612616 2024-05-08T17:05:14.597 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 313532612616 2024-05-08T17:05:14.597 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803834 1-360777252867 2-313532612616' 2024-05-08T17:05:14.597 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T17:05:14.597 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-05-08T17:05:14.746 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=120259084320 2024-05-08T17:05:14.746 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 120259084320 2024-05-08T17:05:14.746 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803834 1-360777252867 2-313532612616 3-120259084320' 2024-05-08T17:05:14.746 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-08T17:05:14.747 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-25769803834 2024-05-08T17:05:14.747 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-08T17:05:14.748 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-05-08T17:05:14.749 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-25769803834 2024-05-08T17:05:14.749 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-08T17:05:14.750 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=25769803834 2024-05-08T17:05:14.750 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 25769803834' 2024-05-08T17:05:14.750 INFO:tasks.workunit.client.0.smithi172.stderr:waiting osd.0 seq 25769803834 2024-05-08T17:05:14.751 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-05-08T17:05:15.138 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 25769803832 -lt 25769803834 2024-05-08T17:05:15.138 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-05-08T17:05:16.140 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-05-08T17:05:16.140 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-05-08T17:05:16.565 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 25769803834 -lt 25769803834 2024-05-08T17:05:16.565 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-08T17:05:16.566 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-360777252867 2024-05-08T17:05:16.566 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-08T17:05:16.567 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-05-08T17:05:16.568 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-360777252867 2024-05-08T17:05:16.568 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-08T17:05:16.569 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=360777252867 2024-05-08T17:05:16.569 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 360777252867' 2024-05-08T17:05:16.569 INFO:tasks.workunit.client.0.smithi172.stderr:waiting osd.1 seq 360777252867 2024-05-08T17:05:16.569 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-05-08T17:05:17.063 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 360777252867 -lt 360777252867 2024-05-08T17:05:17.063 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-08T17:05:17.064 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-313532612616 2024-05-08T17:05:17.064 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-08T17:05:17.065 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-05-08T17:05:17.066 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-313532612616 2024-05-08T17:05:17.066 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-08T17:05:17.067 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=313532612616 2024-05-08T17:05:17.067 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 313532612616' 2024-05-08T17:05:17.067 INFO:tasks.workunit.client.0.smithi172.stderr:waiting osd.2 seq 313532612616 2024-05-08T17:05:17.068 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-05-08T17:05:17.517 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 313532612616 -lt 313532612616 2024-05-08T17:05:17.518 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-08T17:05:17.518 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-120259084320 2024-05-08T17:05:17.518 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-08T17:05:17.519 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-05-08T17:05:17.520 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-120259084320 2024-05-08T17:05:17.520 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-08T17:05:17.521 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=120259084320 2024-05-08T17:05:17.521 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 120259084320' 2024-05-08T17:05:17.521 INFO:tasks.workunit.client.0.smithi172.stderr:waiting osd.3 seq 120259084320 2024-05-08T17:05:17.522 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-05-08T17:05:17.966 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 120259084321 -lt 120259084320 2024-05-08T17:05:17.966 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-05-08T17:05:17.967 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-05-08T17:05:17.967 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-05-08T17:05:18.514 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-05-08T17:05:18.514 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-05-08T17:05:18.515 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-05-08T17:05:18.515 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-05-08T17:05:18.515 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-05-08T17:05:18.515 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-05-08T17:05:18.515 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-05-08T17:05:18.515 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-05-08T17:05:18.922 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-05-08T17:05:18.922 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-05-08T17:05:18.922 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-05-08T17:05:18.923 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-05-08T17:05:19.441 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-05-08T17:05:19.441 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-05-08T17:05:19.441 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-05-08T17:05:19.441 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:209: set_size: dd if=/dev/urandom bs=256 count=1 2024-05-08T17:05:19.446 INFO:tasks.workunit.client.0.smithi172.stderr:1+0 records in 2024-05-08T17:05:19.446 INFO:tasks.workunit.client.0.smithi172.stderr:1+0 records out 2024-05-08T17:05:19.446 INFO:tasks.workunit.client.0.smithi172.stderr:256 bytes copied, 4.4208e-05 s, 5.8 MB/s 2024-05-08T17:05:19.447 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:216: set_size: objectstore_tool td/test-erasure-eio 1 obj-size-81110-1-256 set-bytes td/test-erasure-eio/CORRUPT 2024-05-08T17:05:19.447 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/test-erasure-eio 2024-05-08T17:05:19.447 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-05-08T17:05:19.447 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-05-08T17:05:19.447 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-05-08T17:05:19.447 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/test-erasure-eio 1 obj-size-81110-1-256 set-bytes td/test-erasure-eio/CORRUPT 2024-05-08T17:05:19.447 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/test-erasure-eio 2024-05-08T17:05:19.447 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-05-08T17:05:19.447 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-05-08T17:05:19.447 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-05-08T17:05:19.447 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/test-erasure-eio TERM osd.1 2024-05-08T17:05:19.447 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-08T17:05:19.447 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-08T17:05:19.447 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-08T17:05:19.447 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-08T17:05:19.448 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-08T17:05:20.754 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-08T17:05:20.755 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/test-erasure-eio 1 obj-size-81110-1-256 set-bytes td/test-erasure-eio/CORRUPT 2024-05-08T17:05:20.755 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/test-erasure-eio 2024-05-08T17:05:20.755 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-05-08T17:05:20.755 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-05-08T17:05:20.755 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-05-08T17:05:20.755 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/test-erasure-eio/1 2024-05-08T17:05:20.755 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/test-erasure-eio/1 obj-size-81110-1-256 set-bytes td/test-erasure-eio/CORRUPT 2024-05-08T17:05:22.824 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/test-erasure-eio 1 2024-05-08T17:05:22.824 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/test-erasure-eio 2024-05-08T17:05:22.824 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-05-08T17:05:22.824 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-05-08T17:05:22.824 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-05-08T17:05:22.824 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/test-erasure-eio/1 2024-05-08T17:05:22.824 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=4457b0f1-028b-482b-8ee3-7ee05861e6cc --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-05-08T17:05:22.824 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-08T17:05:22.824 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-05-08T17:05:22.825 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-08T17:05:22.825 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/1' 2024-05-08T17:05:22.825 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/1/journal' 2024-05-08T17:05:22.825 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-05-08T17:05:22.825 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-05-08T17:05:22.825 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-05-08T17:05:22.825 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-05-08T17:05:22.825 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-08T17:05:22.825 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-08T17:05:22.825 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-08T17:05:22.825 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T17:05:22.825 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81110 2024-05-08T17:05:22.827 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.81110/$cluster-$name.asok' 2024-05-08T17:05:22.827 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.81110/$cluster-$name.asok' 2024-05-08T17:05:22.827 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-05-08T17:05:22.827 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-05-08T17:05:22.827 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-05-08T17:05:22.827 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-08T17:05:22.827 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-08T17:05:22.827 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-08T17:05:22.827 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-08T17:05:22.827 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-05-08T17:05:22.827 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-05-08T17:05:22.827 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/test-erasure-eio/1 2024-05-08T17:05:22.828 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-05-08T17:05:22.828 INFO:tasks.workunit.client.0.smithi172.stderr:start osd.1 2024-05-08T17:05:22.828 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=4457b0f1-028b-482b-8ee3-7ee05861e6cc --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81110/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-05-08T17:05:22.828 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/test-erasure-eio/1/whoami 2024-05-08T17:05:22.829 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-05-08T17:05:22.830 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-05-08T17:05:22.830 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-05-08T17:05:22.830 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-05-08T17:05:22.862 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:05:22.865+0000 7f3876b497c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T17:05:22.864 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:05:22.869+0000 7f3876b497c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T17:05:22.865 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:05:22.869+0000 7f3876b497c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T17:05:23.240 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-05-08T17:05:23.240 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-08T17:05:23.240 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-05-08T17:05:23.241 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-08T17:05:23.241 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-08T17:05:23.241 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:05:23.241 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-08T17:05:23.241 INFO:tasks.workunit.client.0.smithi172.stderr:0 2024-05-08T17:05:23.241 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:05:23.241 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T17:05:23.408 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:05:23.413+0000 7f3876b497c0 -1 Falling back to public interface 2024-05-08T17:05:23.705 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:05:24.707 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:05:24.707 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:05:24.707 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-08T17:05:24.707 INFO:tasks.workunit.client.0.smithi172.stderr:1 2024-05-08T17:05:24.708 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:05:24.708 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T17:05:25.105 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:05:26.107 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:05:26.107 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:05:26.108 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-08T17:05:26.108 INFO:tasks.workunit.client.0.smithi172.stderr:2 2024-05-08T17:05:26.108 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:05:26.108 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T17:05:26.185 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:05:26.189+0000 7f3876b497c0 -1 osd.1 86 log_to_monitors true 2024-05-08T17:05:26.783 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:05:27.784 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:05:27.785 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:05:27.785 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-05-08T17:05:27.785 INFO:tasks.workunit.client.0.smithi172.stderr:3 2024-05-08T17:05:27.785 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:05:27.785 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T17:05:28.291 INFO:tasks.workunit.client.0.smithi172.stderr:osd.1 up in weight 1 up_from 91 up_thru 91 down_at 87 last_clean_interval [84,86) [v2:127.0.0.1:6810/3860257724,v1:127.0.0.1:6811/3860257724] [v2:127.0.0.1:6812/3860257724,v1:127.0.0.1:6813/3860257724] exists,up a0a953bf-e981-4e54-97d5-d7c2101753a9 2024-05-08T17:05:28.291 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-08T17:05:28.291 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-08T17:05:28.291 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-08T17:05:28.291 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-05-08T17:05:28.291 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-05-08T17:05:28.291 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-05-08T17:05:28.291 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-05-08T17:05:28.292 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-05-08T17:05:28.292 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-05-08T17:05:28.292 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-05-08T17:05:28.292 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-05-08T17:05:28.293 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-05-08T17:05:28.293 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-05-08T17:05:28.409 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-05-08T17:05:28.409 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-05-08T17:05:28.409 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-05-08T17:05:28.409 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-05-08T17:05:28.409 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-05-08T17:05:28.409 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-05-08T17:05:28.903 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-05-08T17:05:28.903 INFO:tasks.workunit.client.0.smithi172.stderr:1 2024-05-08T17:05:28.903 INFO:tasks.workunit.client.0.smithi172.stderr:2 2024-05-08T17:05:28.903 INFO:tasks.workunit.client.0.smithi172.stderr:3' 2024-05-08T17:05:28.903 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-05-08T17:05:28.903 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T17:05:28.904 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-05-08T17:05:29.075 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=25769803839 2024-05-08T17:05:29.075 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 25769803839 2024-05-08T17:05:29.075 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803839' 2024-05-08T17:05:29.075 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T17:05:29.075 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-05-08T17:05:29.222 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=390842023939 2024-05-08T17:05:29.223 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 390842023939 2024-05-08T17:05:29.223 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803839 1-390842023939' 2024-05-08T17:05:29.223 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T17:05:29.223 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-05-08T17:05:29.375 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=313532612621 2024-05-08T17:05:29.375 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 313532612621 2024-05-08T17:05:29.376 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803839 1-390842023939 2-313532612621' 2024-05-08T17:05:29.376 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T17:05:29.376 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-05-08T17:05:29.534 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=120259084325 2024-05-08T17:05:29.534 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 120259084325 2024-05-08T17:05:29.534 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803839 1-390842023939 2-313532612621 3-120259084325' 2024-05-08T17:05:29.534 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-08T17:05:29.535 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-25769803839 2024-05-08T17:05:29.535 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-08T17:05:29.535 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-05-08T17:05:29.536 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-25769803839 2024-05-08T17:05:29.536 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-08T17:05:29.537 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=25769803839 2024-05-08T17:05:29.537 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 25769803839' 2024-05-08T17:05:29.538 INFO:tasks.workunit.client.0.smithi172.stderr:waiting osd.0 seq 25769803839 2024-05-08T17:05:29.538 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-05-08T17:05:29.960 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 25769803839 -lt 25769803839 2024-05-08T17:05:29.960 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-08T17:05:29.961 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-390842023939 2024-05-08T17:05:29.961 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-08T17:05:29.962 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-05-08T17:05:29.963 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-390842023939 2024-05-08T17:05:29.963 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-08T17:05:29.964 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=390842023939 2024-05-08T17:05:29.964 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 390842023939' 2024-05-08T17:05:29.964 INFO:tasks.workunit.client.0.smithi172.stderr:waiting osd.1 seq 390842023939 2024-05-08T17:05:29.965 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-05-08T17:05:30.443 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 390842023939 -lt 390842023939 2024-05-08T17:05:30.444 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-08T17:05:30.444 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-313532612621 2024-05-08T17:05:30.444 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-08T17:05:30.446 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-05-08T17:05:30.447 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-313532612621 2024-05-08T17:05:30.447 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-08T17:05:30.448 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=313532612621 2024-05-08T17:05:30.448 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 313532612621' 2024-05-08T17:05:30.448 INFO:tasks.workunit.client.0.smithi172.stderr:waiting osd.2 seq 313532612621 2024-05-08T17:05:30.449 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-05-08T17:05:30.861 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 313532612621 -lt 313532612621 2024-05-08T17:05:30.861 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-08T17:05:30.862 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-120259084325 2024-05-08T17:05:30.862 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-08T17:05:30.863 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-05-08T17:05:30.864 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-120259084325 2024-05-08T17:05:30.864 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-08T17:05:30.866 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=120259084325 2024-05-08T17:05:30.866 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 120259084325' 2024-05-08T17:05:30.866 INFO:tasks.workunit.client.0.smithi172.stderr:waiting osd.3 seq 120259084325 2024-05-08T17:05:30.866 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-05-08T17:05:31.285 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 120259084323 -lt 120259084325 2024-05-08T17:05:31.285 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-05-08T17:05:32.286 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-05-08T17:05:32.287 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-05-08T17:05:32.696 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 120259084326 -lt 120259084325 2024-05-08T17:05:32.697 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-05-08T17:05:32.698 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-05-08T17:05:32.698 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-05-08T17:05:33.273 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-05-08T17:05:33.273 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-05-08T17:05:33.274 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-05-08T17:05:33.274 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-05-08T17:05:33.274 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-05-08T17:05:33.274 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-05-08T17:05:33.274 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-05-08T17:05:33.274 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-05-08T17:05:33.694 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-05-08T17:05:33.695 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-05-08T17:05:33.696 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-05-08T17:05:33.696 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-05-08T17:05:34.271 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-05-08T17:05:34.271 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-05-08T17:05:34.271 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-05-08T17:05:34.271 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:217: set_size: rm -f td/test-erasure-eio/CORRUPT 2024-05-08T17:05:34.272 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:218: set_size: ceph osd unset noout 2024-05-08T17:05:34.953 INFO:tasks.workunit.client.0.smithi172.stderr:noout is unset 2024-05-08T17:05:34.975 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:238: rados_get_data_bad_size: rados_get td/test-erasure-eio pool-jerasure obj-size-81110-1-256 2024-05-08T17:05:34.975 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2024-05-08T17:05:34.975 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2024-05-08T17:05:34.975 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=obj-size-81110-1-256 2024-05-08T17:05:34.975 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=ok 2024-05-08T17:05:34.975 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' ok = fail ']' 2024-05-08T17:05:34.975 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:120: rados_get: rados --pool pool-jerasure get obj-size-81110-1-256 td/test-erasure-eio/COPY 2024-05-08T17:05:35.206 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:121: rados_get: diff td/test-erasure-eio/ORIGINAL td/test-erasure-eio/COPY 2024-05-08T17:05:35.208 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:122: rados_get: rm td/test-erasure-eio/COPY 2024-05-08T17:05:35.227 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:241: rados_get_data_bad_size: expr 1 + 1 2024-05-08T17:05:35.227 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:241: rados_get_data_bad_size: shard_id=2 2024-05-08T17:05:35.227 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:242: rados_get_data_bad_size: set_size obj-size-81110-1-256 td/test-erasure-eio 2 256 add 2024-05-08T17:05:35.227 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:192: set_size: local objname=obj-size-81110-1-256 2024-05-08T17:05:35.227 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:193: set_size: shift 2024-05-08T17:05:35.228 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:194: set_size: local dir=td/test-erasure-eio 2024-05-08T17:05:35.228 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:195: set_size: shift 2024-05-08T17:05:35.228 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:196: set_size: local shard_id=2 2024-05-08T17:05:35.228 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:197: set_size: shift 2024-05-08T17:05:35.228 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:198: set_size: local bytes=256 2024-05-08T17:05:35.228 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:199: set_size: shift 2024-05-08T17:05:35.228 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:200: set_size: local mode=add 2024-05-08T17:05:35.228 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:202: set_size: local poolname=pool-jerasure 2024-05-08T17:05:35.228 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: get_osds pool-jerasure obj-size-81110-1-256 2024-05-08T17:05:35.228 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2024-05-08T17:05:35.228 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-size-81110-1-256 2024-05-08T17:05:35.229 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure obj-size-81110-1-256 2024-05-08T17:05:35.229 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-05-08T17:05:35.653 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-05-08T17:05:35.653 INFO:tasks.workunit.client.0.smithi172.stderr:1 2024-05-08T17:05:35.653 INFO:tasks.workunit.client.0.smithi172.stderr:2' 2024-05-08T17:05:35.653 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 2 2024-05-08T17:05:35.653 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: initial_osds=('3' '1' '2') 2024-05-08T17:05:35.653 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:203: set_size: local -a initial_osds 2024-05-08T17:05:35.654 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:204: set_size: local osd_id=2 2024-05-08T17:05:35.654 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:205: set_size: ceph osd set noout 2024-05-08T17:05:36.526 INFO:tasks.workunit.client.0.smithi172.stderr:noout is set 2024-05-08T17:05:36.548 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:206: set_size: '[' add = add ']' 2024-05-08T17:05:36.548 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:208: set_size: objectstore_tool td/test-erasure-eio 2 obj-size-81110-1-256 get-bytes td/test-erasure-eio/CORRUPT 2024-05-08T17:05:36.548 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/test-erasure-eio 2024-05-08T17:05:36.548 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-05-08T17:05:36.548 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=2 2024-05-08T17:05:36.548 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-05-08T17:05:36.548 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/test-erasure-eio 2 obj-size-81110-1-256 get-bytes td/test-erasure-eio/CORRUPT 2024-05-08T17:05:36.548 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/test-erasure-eio 2024-05-08T17:05:36.549 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-05-08T17:05:36.549 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=2 2024-05-08T17:05:36.549 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-05-08T17:05:36.549 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/test-erasure-eio TERM osd.2 2024-05-08T17:05:36.549 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-08T17:05:36.549 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-08T17:05:36.549 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-08T17:05:36.549 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-08T17:05:36.549 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-08T17:05:37.862 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-08T17:05:37.862 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/test-erasure-eio 2 obj-size-81110-1-256 get-bytes td/test-erasure-eio/CORRUPT 2024-05-08T17:05:37.862 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/test-erasure-eio 2024-05-08T17:05:37.862 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-05-08T17:05:37.862 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=2 2024-05-08T17:05:37.862 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-05-08T17:05:37.863 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/test-erasure-eio/2 2024-05-08T17:05:37.863 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/test-erasure-eio/2 obj-size-81110-1-256 get-bytes td/test-erasure-eio/CORRUPT 2024-05-08T17:05:39.798 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/test-erasure-eio 2 2024-05-08T17:05:39.799 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/test-erasure-eio 2024-05-08T17:05:39.799 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-05-08T17:05:39.799 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-05-08T17:05:39.799 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-05-08T17:05:39.799 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/test-erasure-eio/2 2024-05-08T17:05:39.799 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=4457b0f1-028b-482b-8ee3-7ee05861e6cc --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-05-08T17:05:39.800 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-08T17:05:39.800 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-05-08T17:05:39.800 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-08T17:05:39.800 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/2' 2024-05-08T17:05:39.800 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/2/journal' 2024-05-08T17:05:39.806 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-05-08T17:05:39.806 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-05-08T17:05:39.806 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-05-08T17:05:39.806 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-05-08T17:05:39.806 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-08T17:05:39.806 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-08T17:05:39.806 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-08T17:05:39.806 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T17:05:39.806 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81110 2024-05-08T17:05:39.806 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.81110/$cluster-$name.asok' 2024-05-08T17:05:39.807 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.81110/$cluster-$name.asok' 2024-05-08T17:05:39.807 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-05-08T17:05:39.807 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-05-08T17:05:39.807 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-05-08T17:05:39.807 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-08T17:05:39.807 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-08T17:05:39.807 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-08T17:05:39.807 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-08T17:05:39.807 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-05-08T17:05:39.807 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-05-08T17:05:39.807 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/test-erasure-eio/2 2024-05-08T17:05:39.807 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-05-08T17:05:39.807 INFO:tasks.workunit.client.0.smithi172.stderr:start osd.2 2024-05-08T17:05:39.807 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=4457b0f1-028b-482b-8ee3-7ee05861e6cc --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81110/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-05-08T17:05:39.808 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/test-erasure-eio/2/whoami 2024-05-08T17:05:39.808 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-05-08T17:05:39.808 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-05-08T17:05:39.808 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-05-08T17:05:39.808 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-05-08T17:05:39.841 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:05:39.845+0000 7f3e50b987c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T17:05:39.873 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:05:39.877+0000 7f3e50b987c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T17:05:39.874 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:05:39.877+0000 7f3e50b987c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T17:05:40.262 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-05-08T17:05:40.262 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-08T17:05:40.262 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-05-08T17:05:40.262 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-08T17:05:40.262 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-08T17:05:40.262 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:05:40.263 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-08T17:05:40.263 INFO:tasks.workunit.client.0.smithi172.stderr:0 2024-05-08T17:05:40.263 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:05:40.263 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T17:05:40.441 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:05:40.445+0000 7f3e50b987c0 -1 Falling back to public interface 2024-05-08T17:05:40.694 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:05:41.699 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:05:41.700 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:05:41.700 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-08T17:05:41.700 INFO:tasks.workunit.client.0.smithi172.stderr:1 2024-05-08T17:05:41.700 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:05:41.700 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T17:05:42.146 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:05:43.106 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:05:43.109+0000 7f3e50b987c0 -1 osd.2 95 log_to_monitors true 2024-05-08T17:05:43.148 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:05:43.148 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:05:43.148 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-08T17:05:43.148 INFO:tasks.workunit.client.0.smithi172.stderr:2 2024-05-08T17:05:43.148 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T17:05:43.149 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:05:43.679 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:05:44.681 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:05:44.681 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:05:44.681 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-05-08T17:05:44.681 INFO:tasks.workunit.client.0.smithi172.stderr:3 2024-05-08T17:05:44.682 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:05:44.682 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T17:05:45.381 INFO:tasks.workunit.client.0.smithi172.stderr:osd.2 up in weight 1 up_from 101 up_thru 87 down_at 98 last_clean_interval [73,95) [v2:127.0.0.1:6818/3306506109,v1:127.0.0.1:6819/3306506109] [v2:127.0.0.1:6820/3306506109,v1:127.0.0.1:6821/3306506109] exists,up f5e9bb9e-535e-42dd-9e28-654a28c9dc8d 2024-05-08T17:05:45.382 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-08T17:05:45.382 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-08T17:05:45.382 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-08T17:05:45.382 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-05-08T17:05:45.382 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-05-08T17:05:45.382 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-05-08T17:05:45.382 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-05-08T17:05:45.382 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-05-08T17:05:45.383 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-05-08T17:05:45.383 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-05-08T17:05:45.384 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-05-08T17:05:45.384 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-05-08T17:05:45.384 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-05-08T17:05:45.502 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-05-08T17:05:45.502 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-05-08T17:05:45.502 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-05-08T17:05:45.502 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-05-08T17:05:45.502 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-05-08T17:05:45.503 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-05-08T17:05:46.114 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-05-08T17:05:46.114 INFO:tasks.workunit.client.0.smithi172.stderr:1 2024-05-08T17:05:46.114 INFO:tasks.workunit.client.0.smithi172.stderr:2 2024-05-08T17:05:46.114 INFO:tasks.workunit.client.0.smithi172.stderr:3' 2024-05-08T17:05:46.114 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-05-08T17:05:46.115 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T17:05:46.115 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-05-08T17:05:46.288 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=25769803844 2024-05-08T17:05:46.288 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 25769803844 2024-05-08T17:05:46.288 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803844' 2024-05-08T17:05:46.288 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T17:05:46.288 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-05-08T17:05:46.437 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=390842023944 2024-05-08T17:05:46.438 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 390842023944 2024-05-08T17:05:46.438 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803844 1-390842023944' 2024-05-08T17:05:46.438 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T17:05:46.438 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-05-08T17:05:46.595 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=433791696899 2024-05-08T17:05:46.595 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 433791696899 2024-05-08T17:05:46.595 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803844 1-390842023944 2-433791696899' 2024-05-08T17:05:46.596 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T17:05:46.596 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-05-08T17:05:46.749 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=120259084331 2024-05-08T17:05:46.749 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 120259084331 2024-05-08T17:05:46.749 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803844 1-390842023944 2-433791696899 3-120259084331' 2024-05-08T17:05:46.749 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-08T17:05:46.750 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-25769803844 2024-05-08T17:05:46.750 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-08T17:05:46.750 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-05-08T17:05:46.751 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-25769803844 2024-05-08T17:05:46.751 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-08T17:05:46.753 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=25769803844 2024-05-08T17:05:46.753 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 25769803844' 2024-05-08T17:05:46.753 INFO:tasks.workunit.client.0.smithi172.stderr:waiting osd.0 seq 25769803844 2024-05-08T17:05:46.753 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-05-08T17:05:47.154 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 25769803842 -lt 25769803844 2024-05-08T17:05:47.154 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-05-08T17:05:48.155 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-05-08T17:05:48.156 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-05-08T17:05:48.562 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 25769803845 -lt 25769803844 2024-05-08T17:05:48.563 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-08T17:05:48.563 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-390842023944 2024-05-08T17:05:48.564 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-08T17:05:48.566 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-05-08T17:05:48.566 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-390842023944 2024-05-08T17:05:48.567 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-08T17:05:48.567 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=390842023944 2024-05-08T17:05:48.568 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 390842023944' 2024-05-08T17:05:48.568 INFO:tasks.workunit.client.0.smithi172.stderr:waiting osd.1 seq 390842023944 2024-05-08T17:05:48.568 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-05-08T17:05:49.012 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 390842023945 -lt 390842023944 2024-05-08T17:05:49.012 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-08T17:05:49.013 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-433791696899 2024-05-08T17:05:49.013 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-08T17:05:49.015 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-05-08T17:05:49.016 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-433791696899 2024-05-08T17:05:49.016 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-08T17:05:49.017 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=433791696899 2024-05-08T17:05:49.017 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 433791696899' 2024-05-08T17:05:49.017 INFO:tasks.workunit.client.0.smithi172.stderr:waiting osd.2 seq 433791696899 2024-05-08T17:05:49.017 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-05-08T17:05:49.438 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 433791696899 -lt 433791696899 2024-05-08T17:05:49.438 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-08T17:05:49.439 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-120259084331 2024-05-08T17:05:49.439 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-08T17:05:49.441 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-05-08T17:05:49.442 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-120259084331 2024-05-08T17:05:49.442 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-08T17:05:49.443 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=120259084331 2024-05-08T17:05:49.444 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 120259084331' 2024-05-08T17:05:49.444 INFO:tasks.workunit.client.0.smithi172.stderr:waiting osd.3 seq 120259084331 2024-05-08T17:05:49.444 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-05-08T17:05:49.863 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 120259084331 -lt 120259084331 2024-05-08T17:05:49.864 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-05-08T17:05:49.864 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-05-08T17:05:49.864 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-05-08T17:05:50.448 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-05-08T17:05:50.448 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-05-08T17:05:50.448 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-05-08T17:05:50.448 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-05-08T17:05:50.448 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-05-08T17:05:50.448 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-05-08T17:05:50.448 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-05-08T17:05:50.448 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-05-08T17:05:50.828 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-05-08T17:05:50.828 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-05-08T17:05:50.829 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-05-08T17:05:50.829 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-05-08T17:05:51.364 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-05-08T17:05:51.364 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-05-08T17:05:51.364 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-05-08T17:05:51.364 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:209: set_size: dd if=/dev/urandom bs=256 count=1 2024-05-08T17:05:51.366 INFO:tasks.workunit.client.0.smithi172.stderr:1+0 records in 2024-05-08T17:05:51.366 INFO:tasks.workunit.client.0.smithi172.stderr:1+0 records out 2024-05-08T17:05:51.366 INFO:tasks.workunit.client.0.smithi172.stderr:256 bytes copied, 4.1964e-05 s, 6.1 MB/s 2024-05-08T17:05:51.366 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:216: set_size: objectstore_tool td/test-erasure-eio 2 obj-size-81110-1-256 set-bytes td/test-erasure-eio/CORRUPT 2024-05-08T17:05:51.366 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/test-erasure-eio 2024-05-08T17:05:51.366 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-05-08T17:05:51.366 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=2 2024-05-08T17:05:51.366 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-05-08T17:05:51.367 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/test-erasure-eio 2 obj-size-81110-1-256 set-bytes td/test-erasure-eio/CORRUPT 2024-05-08T17:05:51.367 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/test-erasure-eio 2024-05-08T17:05:51.367 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-05-08T17:05:51.367 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=2 2024-05-08T17:05:51.367 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-05-08T17:05:51.367 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/test-erasure-eio TERM osd.2 2024-05-08T17:05:51.367 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-08T17:05:51.367 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-08T17:05:51.367 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-08T17:05:51.367 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-08T17:05:51.367 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-08T17:05:52.677 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-08T17:05:52.677 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/test-erasure-eio 2 obj-size-81110-1-256 set-bytes td/test-erasure-eio/CORRUPT 2024-05-08T17:05:52.677 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/test-erasure-eio 2024-05-08T17:05:52.677 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-05-08T17:05:52.677 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=2 2024-05-08T17:05:52.678 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-05-08T17:05:52.678 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/test-erasure-eio/2 2024-05-08T17:05:52.678 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/test-erasure-eio/2 obj-size-81110-1-256 set-bytes td/test-erasure-eio/CORRUPT 2024-05-08T17:05:54.790 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/test-erasure-eio 2 2024-05-08T17:05:54.791 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/test-erasure-eio 2024-05-08T17:05:54.791 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-05-08T17:05:54.791 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-05-08T17:05:54.791 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-05-08T17:05:54.791 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/test-erasure-eio/2 2024-05-08T17:05:54.791 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=4457b0f1-028b-482b-8ee3-7ee05861e6cc --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-05-08T17:05:54.791 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-08T17:05:54.791 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-05-08T17:05:54.791 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-08T17:05:54.791 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/2' 2024-05-08T17:05:54.791 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/2/journal' 2024-05-08T17:05:54.791 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-05-08T17:05:54.791 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-05-08T17:05:54.791 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-05-08T17:05:54.792 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-05-08T17:05:54.792 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-08T17:05:54.792 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-08T17:05:54.793 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-08T17:05:54.793 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T17:05:54.793 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81110 2024-05-08T17:05:54.793 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.81110/$cluster-$name.asok' 2024-05-08T17:05:54.793 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.81110/$cluster-$name.asok' 2024-05-08T17:05:54.794 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-05-08T17:05:54.794 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-05-08T17:05:54.794 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-05-08T17:05:54.794 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-08T17:05:54.794 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-08T17:05:54.794 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-08T17:05:54.794 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-08T17:05:54.794 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-05-08T17:05:54.794 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-05-08T17:05:54.794 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/test-erasure-eio/2 2024-05-08T17:05:54.795 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-05-08T17:05:54.795 INFO:tasks.workunit.client.0.smithi172.stderr:start osd.2 2024-05-08T17:05:54.795 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=4457b0f1-028b-482b-8ee3-7ee05861e6cc --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81110/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-05-08T17:05:54.795 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/test-erasure-eio/2/whoami 2024-05-08T17:05:54.796 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-05-08T17:05:54.797 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-05-08T17:05:54.797 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-05-08T17:05:54.797 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-05-08T17:05:54.828 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:05:54.833+0000 7fe3c89017c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T17:05:54.831 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:05:54.837+0000 7fe3c89017c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T17:05:54.833 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:05:54.837+0000 7fe3c89017c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T17:05:55.204 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-05-08T17:05:55.205 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-08T17:05:55.205 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-05-08T17:05:55.205 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-08T17:05:55.205 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-08T17:05:55.205 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:05:55.205 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-08T17:05:55.205 INFO:tasks.workunit.client.0.smithi172.stderr:0 2024-05-08T17:05:55.205 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T17:05:55.205 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:05:55.404 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:05:55.409+0000 7fe3c89017c0 -1 Falling back to public interface 2024-05-08T17:05:55.654 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:05:56.656 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:05:56.656 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:05:56.656 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-08T17:05:56.656 INFO:tasks.workunit.client.0.smithi172.stderr:1 2024-05-08T17:05:56.657 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:05:56.657 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T17:05:57.063 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:05:58.065 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:05:58.066 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:05:58.066 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-08T17:05:58.066 INFO:tasks.workunit.client.0.smithi172.stderr:2 2024-05-08T17:05:58.066 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:05:58.066 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T17:05:58.229 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:05:58.232+0000 7fe3c89017c0 -1 osd.2 103 log_to_monitors true 2024-05-08T17:05:58.511 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:05:59.513 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:05:59.514 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:05:59.514 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-05-08T17:05:59.514 INFO:tasks.workunit.client.0.smithi172.stderr:3 2024-05-08T17:05:59.514 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:05:59.514 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T17:06:00.214 INFO:tasks.workunit.client.0.smithi172.stderr:osd.2 up in weight 1 up_from 107 up_thru 87 down_at 104 last_clean_interval [101,103) [v2:127.0.0.1:6818/3822074216,v1:127.0.0.1:6819/3822074216] [v2:127.0.0.1:6820/3822074216,v1:127.0.0.1:6821/3822074216] exists,up f5e9bb9e-535e-42dd-9e28-654a28c9dc8d 2024-05-08T17:06:00.214 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-08T17:06:00.214 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-08T17:06:00.215 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-08T17:06:00.215 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-05-08T17:06:00.215 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-05-08T17:06:00.215 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-05-08T17:06:00.215 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-05-08T17:06:00.216 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-05-08T17:06:00.216 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-05-08T17:06:00.216 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-05-08T17:06:00.216 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-05-08T17:06:00.216 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-05-08T17:06:00.216 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-05-08T17:06:00.326 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-05-08T17:06:00.326 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-05-08T17:06:00.326 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-05-08T17:06:00.326 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-05-08T17:06:00.326 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-05-08T17:06:00.326 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-05-08T17:06:00.846 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-05-08T17:06:00.846 INFO:tasks.workunit.client.0.smithi172.stderr:1 2024-05-08T17:06:00.846 INFO:tasks.workunit.client.0.smithi172.stderr:2 2024-05-08T17:06:00.846 INFO:tasks.workunit.client.0.smithi172.stderr:3' 2024-05-08T17:06:00.846 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-05-08T17:06:00.846 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T17:06:00.846 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-05-08T17:06:01.003 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=25769803849 2024-05-08T17:06:01.003 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 25769803849 2024-05-08T17:06:01.003 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803849' 2024-05-08T17:06:01.003 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T17:06:01.003 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-05-08T17:06:01.152 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=390842023949 2024-05-08T17:06:01.152 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 390842023949 2024-05-08T17:06:01.152 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803849 1-390842023949' 2024-05-08T17:06:01.152 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T17:06:01.152 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-05-08T17:06:01.296 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=459561500675 2024-05-08T17:06:01.297 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 459561500675 2024-05-08T17:06:01.297 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803849 1-390842023949 2-459561500675' 2024-05-08T17:06:01.297 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T17:06:01.297 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-05-08T17:06:01.451 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=120259084336 2024-05-08T17:06:01.451 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 120259084336 2024-05-08T17:06:01.451 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803849 1-390842023949 2-459561500675 3-120259084336' 2024-05-08T17:06:01.451 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-08T17:06:01.452 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-25769803849 2024-05-08T17:06:01.452 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-08T17:06:01.452 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-05-08T17:06:01.453 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-25769803849 2024-05-08T17:06:01.453 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-08T17:06:01.454 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=25769803849 2024-05-08T17:06:01.454 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 25769803849' 2024-05-08T17:06:01.454 INFO:tasks.workunit.client.0.smithi172.stderr:waiting osd.0 seq 25769803849 2024-05-08T17:06:01.454 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-05-08T17:06:01.869 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 25769803849 -lt 25769803849 2024-05-08T17:06:01.870 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-08T17:06:01.871 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-08T17:06:01.871 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-390842023949 2024-05-08T17:06:01.872 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-05-08T17:06:01.873 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-390842023949 2024-05-08T17:06:01.873 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-08T17:06:01.874 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=390842023949 2024-05-08T17:06:01.875 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 390842023949' 2024-05-08T17:06:01.875 INFO:tasks.workunit.client.0.smithi172.stderr:waiting osd.1 seq 390842023949 2024-05-08T17:06:01.875 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-05-08T17:06:02.276 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 390842023949 -lt 390842023949 2024-05-08T17:06:02.276 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-08T17:06:02.277 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-459561500675 2024-05-08T17:06:02.277 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-08T17:06:02.278 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-05-08T17:06:02.279 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-459561500675 2024-05-08T17:06:02.279 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-08T17:06:02.280 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=459561500675 2024-05-08T17:06:02.280 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 459561500675' 2024-05-08T17:06:02.281 INFO:tasks.workunit.client.0.smithi172.stderr:waiting osd.2 seq 459561500675 2024-05-08T17:06:02.281 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-05-08T17:06:02.712 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 459561500675 -lt 459561500675 2024-05-08T17:06:02.712 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-08T17:06:02.713 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-120259084336 2024-05-08T17:06:02.713 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-08T17:06:02.715 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-05-08T17:06:02.715 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-120259084336 2024-05-08T17:06:02.716 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-08T17:06:02.716 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=120259084336 2024-05-08T17:06:02.716 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 120259084336' 2024-05-08T17:06:02.716 INFO:tasks.workunit.client.0.smithi172.stderr:waiting osd.3 seq 120259084336 2024-05-08T17:06:02.717 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-05-08T17:06:03.161 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 120259084334 -lt 120259084336 2024-05-08T17:06:03.161 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-05-08T17:06:04.163 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-05-08T17:06:04.164 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-05-08T17:06:04.585 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 120259084336 -lt 120259084336 2024-05-08T17:06:04.585 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-05-08T17:06:04.586 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-05-08T17:06:04.586 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-05-08T17:06:05.169 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-05-08T17:06:05.169 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-05-08T17:06:05.170 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-05-08T17:06:05.170 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-05-08T17:06:05.170 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-05-08T17:06:05.170 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-05-08T17:06:05.170 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-05-08T17:06:05.171 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-05-08T17:06:05.590 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-05-08T17:06:05.590 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-05-08T17:06:05.590 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-05-08T17:06:05.591 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-05-08T17:06:06.113 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-05-08T17:06:06.114 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-05-08T17:06:06.114 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-05-08T17:06:06.114 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:217: set_size: rm -f td/test-erasure-eio/CORRUPT 2024-05-08T17:06:06.115 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:218: set_size: ceph osd unset noout 2024-05-08T17:06:06.821 INFO:tasks.workunit.client.0.smithi172.stderr:noout is unset 2024-05-08T17:06:06.844 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:243: rados_get_data_bad_size: rados_get td/test-erasure-eio pool-jerasure obj-size-81110-1-256 fail 2024-05-08T17:06:06.844 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2024-05-08T17:06:06.844 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2024-05-08T17:06:06.845 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=obj-size-81110-1-256 2024-05-08T17:06:06.845 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=fail 2024-05-08T17:06:06.845 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' fail = fail ']' 2024-05-08T17:06:06.845 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:114: rados_get: rados --pool pool-jerasure get obj-size-81110-1-256 td/test-erasure-eio/COPY 2024-05-08T17:06:07.037 INFO:tasks.workunit.client.0.smithi172.stderr:error getting pool-jerasure/obj-size-81110-1-256: (5) Input/output error 2024-05-08T17:06:07.040 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:115: rados_get: return 2024-05-08T17:06:07.040 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:244: rados_get_data_bad_size: rm td/test-erasure-eio/ORIGINAL 2024-05-08T17:06:07.042 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:326: TEST_rados_get_bad_size_shard_1: delete_erasure_coded_pool pool-jerasure 2024-05-08T17:06:07.043 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:84: delete_erasure_coded_pool: local poolname=pool-jerasure 2024-05-08T17:06:07.043 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:85: delete_erasure_coded_pool: ceph osd pool delete pool-jerasure pool-jerasure --yes-i-really-really-mean-it 2024-05-08T17:06:07.641 INFO:tasks.workunit.client.0.smithi172.stderr:pool 'pool-jerasure' does not exist 2024-05-08T17:06:07.658 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:86: delete_erasure_coded_pool: ceph osd erasure-code-profile rm myprofile 2024-05-08T17:06:08.538 INFO:tasks.workunit.client.0.smithi172.stderr:erasure-code-profile myprofile does not exist 2024-05-08T17:06:08.557 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:42: run: teardown td/test-erasure-eio 2024-05-08T17:06:08.557 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-eio 2024-05-08T17:06:08.557 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-05-08T17:06:08.557 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-eio KILL 2024-05-08T17:06:08.558 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-08T17:06:08.558 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-08T17:06:08.558 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-08T17:06:08.558 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-08T17:06:08.559 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-08T17:06:08.869 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-08T17:06:08.869 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-05-08T17:06:08.871 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-05-08T17:06:08.872 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-05-08T17:06:08.874 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-05-08T17:06:08.874 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-05-08T17:06:08.874 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-05-08T17:06:08.875 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-08T17:06:08.875 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-05-08T17:06:08.876 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-05-08T17:06:08.876 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-08T17:06:08.877 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-05-08T17:06:08.878 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-05-08T17:06:08.879 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-eio 2024-05-08T17:06:08.920 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-05-08T17:06:08.920 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T17:06:08.920 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81110 2024-05-08T17:06:08.920 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.81110 2024-05-08T17:06:08.922 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-05-08T17:06:08.922 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-05-08T17:06:08.922 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:32: run: for func in $funcs 2024-05-08T17:06:08.922 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:33: run: setup td/test-erasure-eio 2024-05-08T17:06:08.922 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/test-erasure-eio 2024-05-08T17:06:08.922 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/test-erasure-eio 2024-05-08T17:06:08.922 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-eio 2024-05-08T17:06:08.922 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-05-08T17:06:08.922 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-eio KILL 2024-05-08T17:06:08.923 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-08T17:06:08.923 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-08T17:06:08.923 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-08T17:06:08.923 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-08T17:06:08.923 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-08T17:06:08.925 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-08T17:06:08.926 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-05-08T17:06:08.927 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-05-08T17:06:08.927 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-05-08T17:06:08.928 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-05-08T17:06:08.928 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-05-08T17:06:08.929 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-05-08T17:06:08.930 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-08T17:06:08.930 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-05-08T17:06:08.930 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-05-08T17:06:08.931 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-08T17:06:08.932 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-05-08T17:06:08.933 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-05-08T17:06:08.933 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-eio 2024-05-08T17:06:08.935 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-05-08T17:06:08.935 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T17:06:08.935 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81110 2024-05-08T17:06:08.935 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.81110 2024-05-08T17:06:08.936 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-05-08T17:06:08.936 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-05-08T17:06:08.936 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/test-erasure-eio 2024-05-08T17:06:08.938 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-05-08T17:06:08.938 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T17:06:08.938 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81110 2024-05-08T17:06:08.938 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.81110 2024-05-08T17:06:08.940 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-05-08T17:06:08.940 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-05-08T17:06:08.941 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-05-08T17:06:08.941 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/test-erasure-eio 1' TERM HUP INT 2024-05-08T17:06:08.941 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:34: run: run_mon td/test-erasure-eio a 2024-05-08T17:06:08.941 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/test-erasure-eio 2024-05-08T17:06:08.941 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-05-08T17:06:08.941 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-05-08T17:06:08.941 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-05-08T17:06:08.941 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/test-erasure-eio/a 2024-05-08T17:06:08.941 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-eio/a --run-dir=td/test-erasure-eio 2024-05-08T17:06:09.365 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-05-08T17:06:09.365 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-08T17:06:09.366 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-08T17:06:09.366 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-08T17:06:09.366 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T17:06:09.366 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81110 2024-05-08T17:06:09.366 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.81110/$cluster-$name.asok' 2024-05-08T17:06:09.366 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-eio/a '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.81110/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-eio/log --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-05-08T17:06:09.600 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-05-08T17:06:09.601 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-05-08T17:06:09.601 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-05-08T17:06:09.601 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-05-08T17:06:09.601 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-05-08T17:06:09.602 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-05-08T17:06:09.602 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-05-08T17:06:09.602 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-05-08T17:06:09.603 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-05-08T17:06:09.603 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-08T17:06:09.603 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T17:06:09.603 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81110 2024-05-08T17:06:09.603 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.81110/ceph-mon.a.asok 2024-05-08T17:06:09.603 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-05-08T17:06:09.603 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.81110/ceph-mon.a.asok config get fsid 2024-05-08T17:06:09.719 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-05-08T17:06:09.719 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-05-08T17:06:09.719 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-05-08T17:06:09.719 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-05-08T17:06:09.720 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-05-08T17:06:09.720 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-05-08T17:06:09.720 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-05-08T17:06:09.720 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-05-08T17:06:09.720 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-08T17:06:09.720 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T17:06:09.720 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81110 2024-05-08T17:06:09.721 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.81110/ceph-mon.a.asok 2024-05-08T17:06:09.721 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-05-08T17:06:09.721 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.81110/ceph-mon.a.asok config get mon_host 2024-05-08T17:06:09.826 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:35: run: run_mgr td/test-erasure-eio x 2024-05-08T17:06:09.826 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/test-erasure-eio 2024-05-08T17:06:09.826 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-05-08T17:06:09.826 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-05-08T17:06:09.826 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-05-08T17:06:09.826 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/test-erasure-eio/x 2024-05-08T17:06:09.827 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-05-08T17:06:10.751 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-05-08T17:06:10.751 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-08T17:06:10.751 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-08T17:06:10.751 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-08T17:06:10.751 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T17:06:10.751 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81110 2024-05-08T17:06:10.752 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.81110/$cluster-$name.asok' 2024-05-08T17:06:10.752 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-05-08T17:06:10.753 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-eio/x '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.81110/$cluster-$name.asok' --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-05-08T17:06:10.833 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:36: run: create_pool rbd 4 2024-05-08T17:06:10.833 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-05-08T17:06:11.170 INFO:tasks.workunit.client.0.smithi172.stderr:pool 'rbd' already exists 2024-05-08T17:06:11.186 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-05-08T17:06:12.190 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: get_asok_path mon.a 2024-05-08T17:06:12.190 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-05-08T17:06:12.190 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-05-08T17:06:12.190 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-08T17:06:12.190 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T17:06:12.191 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81110 2024-05-08T17:06:12.191 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.81110/ceph-mon.a.asok 2024-05-08T17:06:12.191 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: CEPH_ARGS= 2024-05-08T17:06:12.191 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: ceph --admin-daemon /tmp/ceph-asok.81110/ceph-mon.a.asok log flush 2024-05-08T17:06:12.291 INFO:tasks.workunit.client.0.smithi172.stdout:{} 2024-05-08T17:06:12.305 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:40: run: grep 'load: jerasure.*lrc' td/test-erasure-eio/mon.a.log 2024-05-08T17:06:12.307 INFO:tasks.workunit.client.0.smithi172.stdout:2024-05-08T17:06:09.412+0000 7fc05cb0ec80 0 load: jerasure load: lrc 2024-05-08T17:06:12.308 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:41: run: TEST_rados_get_subread_eio_shard_0 td/test-erasure-eio 2024-05-08T17:06:12.308 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:255: TEST_rados_get_subread_eio_shard_0: local dir=td/test-erasure-eio 2024-05-08T17:06:12.308 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:256: TEST_rados_get_subread_eio_shard_0: setup_osds 4 2024-05-08T17:06:12.309 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:47: setup_osds: local count=4 2024-05-08T17:06:12.309 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:48: setup_osds: shift 2024-05-08T17:06:12.309 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: expr 4 - 1 2024-05-08T17:06:12.311 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: seq 0 3 2024-05-08T17:06:12.313 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-05-08T17:06:12.313 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 0 2024-05-08T17:06:12.313 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-05-08T17:06:12.313 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-05-08T17:06:12.314 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-05-08T17:06:12.314 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-05-08T17:06:12.314 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/0 2024-05-08T17:06:12.314 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=4457b0f1-028b-482b-8ee3-7ee05861e6cc --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-05-08T17:06:12.314 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-08T17:06:12.314 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-05-08T17:06:12.314 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-08T17:06:12.314 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/0' 2024-05-08T17:06:12.314 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/0/journal' 2024-05-08T17:06:12.314 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-05-08T17:06:12.314 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-05-08T17:06:12.314 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-05-08T17:06:12.315 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-05-08T17:06:12.315 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-08T17:06:12.315 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-08T17:06:12.315 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-08T17:06:12.315 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T17:06:12.315 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81110 2024-05-08T17:06:12.315 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.81110/$cluster-$name.asok' 2024-05-08T17:06:12.316 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.81110/$cluster-$name.asok' 2024-05-08T17:06:12.317 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-05-08T17:06:12.317 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-05-08T17:06:12.317 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-05-08T17:06:12.317 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-05-08T17:06:12.317 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-05-08T17:06:12.317 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-08T17:06:12.317 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-08T17:06:12.317 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-08T17:06:12.317 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-08T17:06:12.317 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-05-08T17:06:12.317 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-05-08T17:06:12.317 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/0 2024-05-08T17:06:12.317 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-05-08T17:06:12.317 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=9e94024d-fc60-4ed6-bf9f-81627333e8cc 2024-05-08T17:06:12.317 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 9e94024d-fc60-4ed6-bf9f-81627333e8cc' 2024-05-08T17:06:12.318 INFO:tasks.workunit.client.0.smithi172.stdout:add osd0 9e94024d-fc60-4ed6-bf9f-81627333e8cc 2024-05-08T17:06:12.318 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-05-08T17:06:12.335 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAEsTtmCPtlFBAA4Bi+QloA/Ljfp53IO7DgYA== 2024-05-08T17:06:12.335 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAEsTtmCPtlFBAA4Bi+QloA/Ljfp53IO7DgYA=="}' 2024-05-08T17:06:12.335 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 9e94024d-fc60-4ed6-bf9f-81627333e8cc -i td/test-erasure-eio/0/new.json 2024-05-08T17:06:12.647 INFO:tasks.workunit.client.0.smithi172.stdout:0 2024-05-08T17:06:12.662 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/0/new.json 2024-05-08T17:06:12.663 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=4457b0f1-028b-482b-8ee3-7ee05861e6cc --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81110/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAEsTtmCPtlFBAA4Bi+QloA/Ljfp53IO7DgYA== --osd-uuid 9e94024d-fc60-4ed6-bf9f-81627333e8cc 2024-05-08T17:06:12.699 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:06:12.700+0000 7fa5220027c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T17:06:12.724 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:06:12.728+0000 7fa5220027c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T17:06:12.725 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:06:12.728+0000 7fa5220027c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T17:06:12.725 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:06:12.728+0000 7fa5220027c0 -1 bluestore(td/test-erasure-eio/0/block) _read_bdev_label failed to open td/test-erasure-eio/0/block: (2) No such file or directory 2024-05-08T17:06:12.725 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:06:12.728+0000 7fa5220027c0 -1 bluestore(td/test-erasure-eio/0/block) _read_bdev_label failed to open td/test-erasure-eio/0/block: (2) No such file or directory 2024-05-08T17:06:12.725 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:06:12.728+0000 7fa5220027c0 -1 bluestore(td/test-erasure-eio/0/block) _read_bdev_label failed to open td/test-erasure-eio/0/block: (2) No such file or directory 2024-05-08T17:06:12.763 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:06:12.768+0000 7fa5220027c0 -1 bluestore(td/test-erasure-eio/0) _read_fsid unparsable uuid 2024-05-08T17:06:19.440 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/0/keyring 2024-05-08T17:06:19.441 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-05-08T17:06:19.442 INFO:tasks.workunit.client.0.smithi172.stdout:adding osd0 key to auth repository 2024-05-08T17:06:19.442 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-05-08T17:06:19.442 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-05-08T17:06:19.990 INFO:tasks.workunit.client.0.smithi172.stdout:start osd.0 2024-05-08T17:06:19.991 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-05-08T17:06:19.991 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=4457b0f1-028b-482b-8ee3-7ee05861e6cc --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81110/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-05-08T17:06:19.991 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-05-08T17:06:19.991 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-05-08T17:06:19.991 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-05-08T17:06:20.022 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:06:20.024+0000 7fb699b7f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T17:06:20.062 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:06:20.064+0000 7fb699b7f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T17:06:20.063 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:06:20.068+0000 7fb699b7f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T17:06:20.432 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-05-08T17:06:20.432 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-08T17:06:20.432 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-05-08T17:06:20.432 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-08T17:06:20.432 INFO:tasks.workunit.client.0.smithi172.stdout:0 2024-05-08T17:06:20.433 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-08T17:06:20.433 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:06:20.433 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-08T17:06:20.433 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:06:20.433 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T17:06:20.610 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:06:20.612+0000 7fb699b7f7c0 -1 Falling back to public interface 2024-05-08T17:06:20.839 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:06:21.840 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:06:21.841 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:06:21.841 INFO:tasks.workunit.client.0.smithi172.stdout:1 2024-05-08T17:06:21.841 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-08T17:06:21.842 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:06:21.842 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T17:06:22.253 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:06:23.176 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:06:23.180+0000 7fb699b7f7c0 -1 osd.0 0 log_to_monitors true 2024-05-08T17:06:23.255 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:06:23.255 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:06:23.255 INFO:tasks.workunit.client.0.smithi172.stdout:2 2024-05-08T17:06:23.256 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-08T17:06:23.256 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:06:23.256 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T17:06:23.903 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:06:24.905 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:06:24.905 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:06:24.905 INFO:tasks.workunit.client.0.smithi172.stdout:3 2024-05-08T17:06:24.906 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-05-08T17:06:24.906 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:06:24.906 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T17:06:25.384 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:06:26.386 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:06:26.386 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:06:26.386 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-05-08T17:06:26.387 INFO:tasks.workunit.client.0.smithi172.stdout:4 2024-05-08T17:06:26.387 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:06:26.387 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T17:06:26.806 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:06:27.808 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:06:27.809 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:06:27.809 INFO:tasks.workunit.client.0.smithi172.stdout:5 2024-05-08T17:06:27.809 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-05-08T17:06:27.809 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:06:27.810 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T17:06:28.475 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:06:29.477 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:06:29.477 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:06:29.478 INFO:tasks.workunit.client.0.smithi172.stdout:6 2024-05-08T17:06:29.478 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-05-08T17:06:29.478 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:06:29.478 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T17:06:30.035 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:06:31.036 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:06:31.037 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:06:31.037 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-05-08T17:06:31.037 INFO:tasks.workunit.client.0.smithi172.stdout:7 2024-05-08T17:06:31.037 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:06:31.037 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T17:06:31.463 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:06:32.464 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:06:32.464 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:06:32.464 INFO:tasks.workunit.client.0.smithi172.stdout:8 2024-05-08T17:06:32.465 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-05-08T17:06:32.465 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:06:32.465 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T17:06:33.032 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:06:34.033 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:06:34.033 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:06:34.034 INFO:tasks.workunit.client.0.smithi172.stdout:9 2024-05-08T17:06:34.034 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-05-08T17:06:34.034 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:06:34.035 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T17:06:34.508 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:06:35.509 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:06:35.510 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:06:35.510 INFO:tasks.workunit.client.0.smithi172.stdout:10 2024-05-08T17:06:35.510 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-05-08T17:06:35.511 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:06:35.511 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T17:06:36.071 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:06:37.070 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:06:37.071 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:06:37.071 INFO:tasks.workunit.client.0.smithi172.stdout:11 2024-05-08T17:06:37.071 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-05-08T17:06:37.072 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:06:37.072 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T17:06:37.605 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:06:38.607 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:06:38.607 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:06:38.607 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-05-08T17:06:38.608 INFO:tasks.workunit.client.0.smithi172.stdout:12 2024-05-08T17:06:38.608 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:06:38.608 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T17:06:39.359 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:06:40.360 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:06:40.360 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:06:40.361 INFO:tasks.workunit.client.0.smithi172.stdout:13 2024-05-08T17:06:40.361 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-05-08T17:06:40.361 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:06:40.361 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T17:06:41.115 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:06:42.116 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:06:42.116 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:06:42.117 INFO:tasks.workunit.client.0.smithi172.stdout:14 2024-05-08T17:06:42.117 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-05-08T17:06:42.117 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:06:42.117 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T17:06:43.296 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:06:44.298 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:06:44.298 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:06:44.298 INFO:tasks.workunit.client.0.smithi172.stdout:15 2024-05-08T17:06:44.299 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-05-08T17:06:44.299 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:06:44.299 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T17:06:45.245 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:06:46.247 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:06:46.247 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:06:46.247 INFO:tasks.workunit.client.0.smithi172.stdout:16 2024-05-08T17:06:46.248 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-05-08T17:06:46.248 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:06:46.248 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T17:06:47.457 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:06:48.459 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:06:48.460 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:06:48.460 INFO:tasks.workunit.client.0.smithi172.stdout:17 2024-05-08T17:06:48.460 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-05-08T17:06:48.460 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:06:48.460 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T17:06:48.932 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:06:49.933 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:06:49.933 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:06:49.934 INFO:tasks.workunit.client.0.smithi172.stdout:18 2024-05-08T17:06:49.934 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-05-08T17:06:49.934 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:06:49.934 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T17:06:50.407 INFO:tasks.workunit.client.0.smithi172.stdout:osd.0 up in weight 1 up_from 6 up_thru 6 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1413384987,v1:127.0.0.1:6803/1413384987] [v2:127.0.0.1:6804/1413384987,v1:127.0.0.1:6805/1413384987] exists,up 9e94024d-fc60-4ed6-bf9f-81627333e8cc 2024-05-08T17:06:50.409 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-08T17:06:50.409 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-08T17:06:50.410 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-08T17:06:50.410 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-05-08T17:06:50.410 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 1 2024-05-08T17:06:50.410 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-05-08T17:06:50.410 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-05-08T17:06:50.410 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-05-08T17:06:50.410 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-05-08T17:06:50.410 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/1 2024-05-08T17:06:50.410 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=4457b0f1-028b-482b-8ee3-7ee05861e6cc --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-05-08T17:06:50.410 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-08T17:06:50.410 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-05-08T17:06:50.410 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-08T17:06:50.410 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/1' 2024-05-08T17:06:50.411 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/1/journal' 2024-05-08T17:06:50.411 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-05-08T17:06:50.411 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-05-08T17:06:50.411 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-05-08T17:06:50.411 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-05-08T17:06:50.411 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-08T17:06:50.411 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-08T17:06:50.413 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-08T17:06:50.413 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T17:06:50.413 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81110 2024-05-08T17:06:50.413 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.81110/$cluster-$name.asok' 2024-05-08T17:06:50.413 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.81110/$cluster-$name.asok' 2024-05-08T17:06:50.413 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-05-08T17:06:50.413 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-05-08T17:06:50.413 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-05-08T17:06:50.413 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-05-08T17:06:50.413 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-05-08T17:06:50.414 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-08T17:06:50.414 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-08T17:06:50.414 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-08T17:06:50.414 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-08T17:06:50.414 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-05-08T17:06:50.414 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-05-08T17:06:50.414 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/1 2024-05-08T17:06:50.414 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-05-08T17:06:50.415 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=1d7905b1-f71f-4b0d-8f66-c79227771371 2024-05-08T17:06:50.415 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 1d7905b1-f71f-4b0d-8f66-c79227771371' 2024-05-08T17:06:50.415 INFO:tasks.workunit.client.0.smithi172.stdout:add osd1 1d7905b1-f71f-4b0d-8f66-c79227771371 2024-05-08T17:06:50.415 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-05-08T17:06:50.432 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAqsTtmG7kxGhAABemmgQslLHq+tnSoiwfOhQ== 2024-05-08T17:06:50.432 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAqsTtmG7kxGhAABemmgQslLHq+tnSoiwfOhQ=="}' 2024-05-08T17:06:50.432 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 1d7905b1-f71f-4b0d-8f66-c79227771371 -i td/test-erasure-eio/1/new.json 2024-05-08T17:06:50.921 INFO:tasks.workunit.client.0.smithi172.stdout:1 2024-05-08T17:06:50.937 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/1/new.json 2024-05-08T17:06:50.938 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=4457b0f1-028b-482b-8ee3-7ee05861e6cc --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81110/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAqsTtmG7kxGhAABemmgQslLHq+tnSoiwfOhQ== --osd-uuid 1d7905b1-f71f-4b0d-8f66-c79227771371 2024-05-08T17:06:50.971 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:06:50.975+0000 7faf0643e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T17:06:51.018 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:06:51.019+0000 7faf0643e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T17:06:51.020 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:06:51.023+0000 7faf0643e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T17:06:51.020 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:06:51.023+0000 7faf0643e7c0 -1 bluestore(td/test-erasure-eio/1/block) _read_bdev_label failed to open td/test-erasure-eio/1/block: (2) No such file or directory 2024-05-08T17:06:51.020 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:06:51.023+0000 7faf0643e7c0 -1 bluestore(td/test-erasure-eio/1/block) _read_bdev_label failed to open td/test-erasure-eio/1/block: (2) No such file or directory 2024-05-08T17:06:51.021 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:06:51.023+0000 7faf0643e7c0 -1 bluestore(td/test-erasure-eio/1/block) _read_bdev_label failed to open td/test-erasure-eio/1/block: (2) No such file or directory 2024-05-08T17:06:51.065 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:06:51.067+0000 7faf0643e7c0 -1 bluestore(td/test-erasure-eio/1) _read_fsid unparsable uuid 2024-05-08T17:06:57.876 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/1/keyring 2024-05-08T17:06:57.876 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-05-08T17:06:57.877 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-05-08T17:06:57.878 INFO:tasks.workunit.client.0.smithi172.stdout:adding osd1 key to auth repository 2024-05-08T17:06:57.878 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-05-08T17:06:58.439 INFO:tasks.workunit.client.0.smithi172.stdout:start osd.1 2024-05-08T17:06:58.440 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-05-08T17:06:58.440 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=4457b0f1-028b-482b-8ee3-7ee05861e6cc --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81110/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-05-08T17:06:58.440 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-05-08T17:06:58.440 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-05-08T17:06:58.440 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-05-08T17:06:58.471 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:06:58.479+0000 7f91fb2827c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T17:06:58.473 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:06:58.479+0000 7f91fb2827c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T17:06:58.474 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:06:58.479+0000 7f91fb2827c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T17:06:58.836 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-05-08T17:06:58.836 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-08T17:06:58.836 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-05-08T17:06:58.836 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-08T17:06:58.836 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-08T17:06:58.837 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:06:58.837 INFO:tasks.workunit.client.0.smithi172.stdout:0 2024-05-08T17:06:58.837 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-08T17:06:58.837 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T17:06:58.837 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:06:59.034 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:06:59.039+0000 7f91fb2827c0 -1 Falling back to public interface 2024-05-08T17:06:59.292 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:07:00.294 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:07:00.294 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:07:00.295 INFO:tasks.workunit.client.0.smithi172.stdout:1 2024-05-08T17:07:00.295 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-08T17:07:00.295 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:07:00.295 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T17:07:00.700 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:07:01.702 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:07:01.702 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:07:01.702 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-08T17:07:01.702 INFO:tasks.workunit.client.0.smithi172.stdout:2 2024-05-08T17:07:01.703 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:07:01.703 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T17:07:01.780 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:07:01.787+0000 7f91fb2827c0 -1 osd.1 0 log_to_monitors true 2024-05-08T17:07:02.172 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:07:03.174 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:07:03.174 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:07:03.175 INFO:tasks.workunit.client.0.smithi172.stdout:3 2024-05-08T17:07:03.175 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-05-08T17:07:03.175 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:07:03.176 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T17:07:03.810 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:07:04.812 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:07:04.812 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:07:04.812 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-05-08T17:07:04.812 INFO:tasks.workunit.client.0.smithi172.stdout:4 2024-05-08T17:07:04.813 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:07:04.813 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T17:07:05.274 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:07:06.275 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:07:06.276 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:07:06.276 INFO:tasks.workunit.client.0.smithi172.stdout:5 2024-05-08T17:07:06.276 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-05-08T17:07:06.277 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:07:06.277 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T17:07:06.741 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:07:07.743 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:07:07.743 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:07:07.744 INFO:tasks.workunit.client.0.smithi172.stdout:6 2024-05-08T17:07:07.744 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-05-08T17:07:07.744 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:07:07.744 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T17:07:08.211 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:07:09.213 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:07:09.213 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:07:09.213 INFO:tasks.workunit.client.0.smithi172.stdout:7 2024-05-08T17:07:09.214 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-05-08T17:07:09.214 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:07:09.214 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T17:07:09.765 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:07:10.767 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:07:10.767 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:07:10.767 INFO:tasks.workunit.client.0.smithi172.stdout:8 2024-05-08T17:07:10.768 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-05-08T17:07:10.768 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:07:10.768 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T17:07:11.219 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:07:12.222 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:07:12.222 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:07:12.222 INFO:tasks.workunit.client.0.smithi172.stdout:9 2024-05-08T17:07:12.223 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-05-08T17:07:12.223 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:07:12.223 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T17:07:12.717 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:07:13.718 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:07:13.718 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:07:13.719 INFO:tasks.workunit.client.0.smithi172.stdout:10 2024-05-08T17:07:13.719 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-05-08T17:07:13.719 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:07:13.719 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T17:07:14.170 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:07:15.171 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:07:15.172 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:07:15.172 INFO:tasks.workunit.client.0.smithi172.stdout:11 2024-05-08T17:07:15.172 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-05-08T17:07:15.173 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:07:15.173 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T17:07:15.807 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:07:16.809 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:07:16.809 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:07:16.809 INFO:tasks.workunit.client.0.smithi172.stdout:12 2024-05-08T17:07:16.810 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-05-08T17:07:16.810 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:07:16.810 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T17:07:17.704 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:07:18.706 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:07:18.706 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:07:18.706 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-05-08T17:07:18.706 INFO:tasks.workunit.client.0.smithi172.stdout:13 2024-05-08T17:07:18.706 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:07:18.707 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T17:07:19.607 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:07:20.609 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:07:20.609 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:07:20.610 INFO:tasks.workunit.client.0.smithi172.stdout:14 2024-05-08T17:07:20.610 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-05-08T17:07:20.610 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:07:20.610 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T17:07:21.587 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:07:22.589 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:07:22.589 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:07:22.589 INFO:tasks.workunit.client.0.smithi172.stdout:15 2024-05-08T17:07:22.590 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-05-08T17:07:22.590 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:07:22.590 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T17:07:23.443 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:07:24.444 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:07:24.444 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:07:24.445 INFO:tasks.workunit.client.0.smithi172.stdout:16 2024-05-08T17:07:24.445 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-05-08T17:07:24.445 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:07:24.446 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T17:07:25.298 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:07:26.300 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:07:26.300 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:07:26.300 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-05-08T17:07:26.300 INFO:tasks.workunit.client.0.smithi172.stdout:17 2024-05-08T17:07:26.301 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:07:26.301 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T17:07:27.221 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:07:28.223 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:07:28.224 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:07:28.224 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-05-08T17:07:28.224 INFO:tasks.workunit.client.0.smithi172.stdout:18 2024-05-08T17:07:28.224 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:07:28.224 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T17:07:28.674 INFO:tasks.workunit.client.0.smithi172.stdout:osd.1 up in weight 1 up_from 13 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/1292646229,v1:127.0.0.1:6811/1292646229] [v2:127.0.0.1:6812/1292646229,v1:127.0.0.1:6813/1292646229] exists,up 1d7905b1-f71f-4b0d-8f66-c79227771371 2024-05-08T17:07:28.680 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-08T17:07:28.680 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-08T17:07:28.680 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-08T17:07:28.680 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-05-08T17:07:28.680 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 2 2024-05-08T17:07:28.680 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-05-08T17:07:28.680 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-05-08T17:07:28.680 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-05-08T17:07:28.681 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-05-08T17:07:28.681 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/2 2024-05-08T17:07:28.681 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=4457b0f1-028b-482b-8ee3-7ee05861e6cc --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-05-08T17:07:28.681 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-08T17:07:28.681 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-05-08T17:07:28.681 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-08T17:07:28.681 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/2' 2024-05-08T17:07:28.681 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/2/journal' 2024-05-08T17:07:28.681 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-05-08T17:07:28.682 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-05-08T17:07:28.682 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-05-08T17:07:28.682 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-05-08T17:07:28.682 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-08T17:07:28.682 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-08T17:07:28.682 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-08T17:07:28.682 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T17:07:28.682 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81110 2024-05-08T17:07:28.682 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.81110/$cluster-$name.asok' 2024-05-08T17:07:28.682 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.81110/$cluster-$name.asok' 2024-05-08T17:07:28.682 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-05-08T17:07:28.682 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-05-08T17:07:28.682 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-05-08T17:07:28.683 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-05-08T17:07:28.683 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-05-08T17:07:28.683 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-08T17:07:28.683 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-08T17:07:28.683 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-08T17:07:28.683 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-08T17:07:28.683 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-05-08T17:07:28.683 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-05-08T17:07:28.683 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/2 2024-05-08T17:07:28.683 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-05-08T17:07:28.684 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=2d2fdcef-2483-4ffe-a3b3-faed9797fd1f 2024-05-08T17:07:28.684 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 2d2fdcef-2483-4ffe-a3b3-faed9797fd1f' 2024-05-08T17:07:28.684 INFO:tasks.workunit.client.0.smithi172.stdout:add osd2 2d2fdcef-2483-4ffe-a3b3-faed9797fd1f 2024-05-08T17:07:28.684 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-05-08T17:07:28.703 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBQsTtmDr9UKhAAKfIyWIsm2/fgntRTJ0vrbQ== 2024-05-08T17:07:28.703 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBQsTtmDr9UKhAAKfIyWIsm2/fgntRTJ0vrbQ=="}' 2024-05-08T17:07:28.703 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 2d2fdcef-2483-4ffe-a3b3-faed9797fd1f -i td/test-erasure-eio/2/new.json 2024-05-08T17:07:30.302 INFO:tasks.workunit.client.0.smithi172.stdout:2 2024-05-08T17:07:30.322 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/2/new.json 2024-05-08T17:07:30.324 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=4457b0f1-028b-482b-8ee3-7ee05861e6cc --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81110/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBQsTtmDr9UKhAAKfIyWIsm2/fgntRTJ0vrbQ== --osd-uuid 2d2fdcef-2483-4ffe-a3b3-faed9797fd1f 2024-05-08T17:07:30.353 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:07:30.358+0000 7f24faf547c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T17:07:30.408 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:07:30.414+0000 7f24faf547c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T17:07:30.410 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:07:30.414+0000 7f24faf547c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T17:07:30.410 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:07:30.414+0000 7f24faf547c0 -1 bluestore(td/test-erasure-eio/2/block) _read_bdev_label failed to open td/test-erasure-eio/2/block: (2) No such file or directory 2024-05-08T17:07:30.410 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:07:30.414+0000 7f24faf547c0 -1 bluestore(td/test-erasure-eio/2/block) _read_bdev_label failed to open td/test-erasure-eio/2/block: (2) No such file or directory 2024-05-08T17:07:30.410 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:07:30.414+0000 7f24faf547c0 -1 bluestore(td/test-erasure-eio/2/block) _read_bdev_label failed to open td/test-erasure-eio/2/block: (2) No such file or directory 2024-05-08T17:07:30.508 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:07:30.514+0000 7f24faf547c0 -1 bluestore(td/test-erasure-eio/2) _read_fsid unparsable uuid 2024-05-08T17:07:37.328 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/2/keyring 2024-05-08T17:07:37.329 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-05-08T17:07:37.330 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-05-08T17:07:37.330 INFO:tasks.workunit.client.0.smithi172.stdout:adding osd2 key to auth repository 2024-05-08T17:07:37.331 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-05-08T17:07:37.891 INFO:tasks.workunit.client.0.smithi172.stdout:start osd.2 2024-05-08T17:07:37.892 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-05-08T17:07:37.892 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=4457b0f1-028b-482b-8ee3-7ee05861e6cc --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81110/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-05-08T17:07:37.892 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-05-08T17:07:37.892 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-05-08T17:07:37.892 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-05-08T17:07:37.923 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:07:37.930+0000 7fa6c6fde7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T17:07:37.925 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:07:37.930+0000 7fa6c6fde7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T17:07:37.927 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:07:37.930+0000 7fa6c6fde7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T17:07:38.360 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-05-08T17:07:38.360 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-08T17:07:38.360 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-05-08T17:07:38.360 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-08T17:07:38.360 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-08T17:07:38.360 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:07:38.361 INFO:tasks.workunit.client.0.smithi172.stdout:0 2024-05-08T17:07:38.361 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-08T17:07:38.361 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:07:38.361 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T17:07:38.478 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:07:38.482+0000 7fa6c6fde7c0 -1 Falling back to public interface 2024-05-08T17:07:38.782 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:07:39.783 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:07:39.783 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:07:39.784 INFO:tasks.workunit.client.0.smithi172.stdout:1 2024-05-08T17:07:39.784 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-08T17:07:39.784 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:07:39.784 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T17:07:40.226 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:07:41.140 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:07:41.146+0000 7fa6c6fde7c0 -1 osd.2 0 log_to_monitors true 2024-05-08T17:07:41.228 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:07:41.228 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:07:41.228 INFO:tasks.workunit.client.0.smithi172.stdout:2 2024-05-08T17:07:41.229 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-08T17:07:41.229 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:07:41.229 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T17:07:41.939 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:07:42.940 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:07:42.941 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:07:42.941 INFO:tasks.workunit.client.0.smithi172.stdout:3 2024-05-08T17:07:42.941 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-05-08T17:07:42.941 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:07:42.941 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T17:07:43.431 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:07:44.433 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:07:44.433 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:07:44.433 INFO:tasks.workunit.client.0.smithi172.stdout:4 2024-05-08T17:07:44.434 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-05-08T17:07:44.434 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:07:44.434 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T17:07:44.847 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:07:45.849 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:07:45.849 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:07:45.849 INFO:tasks.workunit.client.0.smithi172.stdout:5 2024-05-08T17:07:45.850 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-05-08T17:07:45.850 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:07:45.850 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T17:07:46.378 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:07:47.380 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:07:47.380 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:07:47.380 INFO:tasks.workunit.client.0.smithi172.stdout:6 2024-05-08T17:07:47.381 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-05-08T17:07:47.381 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:07:47.381 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T17:07:47.852 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:07:48.855 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:07:48.855 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:07:48.855 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-05-08T17:07:48.856 INFO:tasks.workunit.client.0.smithi172.stdout:7 2024-05-08T17:07:48.856 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:07:48.856 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T17:07:49.388 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:07:50.390 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:07:50.391 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:07:50.391 INFO:tasks.workunit.client.0.smithi172.stdout:8 2024-05-08T17:07:50.392 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-05-08T17:07:50.392 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:07:50.392 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T17:07:50.840 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:07:51.842 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:07:51.842 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:07:51.842 INFO:tasks.workunit.client.0.smithi172.stdout:9 2024-05-08T17:07:51.843 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-05-08T17:07:51.843 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:07:51.843 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T17:07:52.364 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:07:53.366 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:07:53.366 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:07:53.366 INFO:tasks.workunit.client.0.smithi172.stdout:10 2024-05-08T17:07:53.367 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-05-08T17:07:53.367 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:07:53.367 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T17:07:53.807 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:07:54.809 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:07:54.809 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:07:54.810 INFO:tasks.workunit.client.0.smithi172.stdout:11 2024-05-08T17:07:54.810 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-05-08T17:07:54.810 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:07:54.810 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T17:07:55.450 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:07:56.451 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:07:56.452 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:07:56.452 INFO:tasks.workunit.client.0.smithi172.stdout:12 2024-05-08T17:07:56.452 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-05-08T17:07:56.453 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:07:56.453 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T17:07:57.479 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:07:58.480 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:07:58.481 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:07:58.481 INFO:tasks.workunit.client.0.smithi172.stdout:13 2024-05-08T17:07:58.481 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-05-08T17:07:58.481 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:07:58.482 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T17:07:59.264 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:08:00.266 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:08:00.266 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:08:00.267 INFO:tasks.workunit.client.0.smithi172.stdout:14 2024-05-08T17:08:00.267 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-05-08T17:08:00.267 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:08:00.267 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T17:08:01.261 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:08:02.263 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:08:02.264 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:08:02.264 INFO:tasks.workunit.client.0.smithi172.stdout:15 2024-05-08T17:08:02.264 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-05-08T17:08:02.264 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:08:02.265 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T17:08:03.063 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:08:04.065 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:08:04.065 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:08:04.066 INFO:tasks.workunit.client.0.smithi172.stdout:16 2024-05-08T17:08:04.067 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-05-08T17:08:04.067 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:08:04.067 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T17:08:05.155 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:08:06.157 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:08:06.157 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:08:06.157 INFO:tasks.workunit.client.0.smithi172.stdout:17 2024-05-08T17:08:06.157 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-05-08T17:08:06.158 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:08:06.158 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T17:08:06.800 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:08:07.802 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:08:07.803 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:08:07.803 INFO:tasks.workunit.client.0.smithi172.stdout:18 2024-05-08T17:08:07.803 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-05-08T17:08:07.803 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:08:07.803 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T17:08:08.225 INFO:tasks.workunit.client.0.smithi172.stdout:osd.2 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/2775383342,v1:127.0.0.1:6819/2775383342] [v2:127.0.0.1:6820/2775383342,v1:127.0.0.1:6821/2775383342] exists,up 2d2fdcef-2483-4ffe-a3b3-faed9797fd1f 2024-05-08T17:08:08.230 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-08T17:08:08.230 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-08T17:08:08.230 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-08T17:08:08.230 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-05-08T17:08:08.230 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 3 2024-05-08T17:08:08.230 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-05-08T17:08:08.230 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-05-08T17:08:08.230 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2024-05-08T17:08:08.230 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-05-08T17:08:08.231 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/3 2024-05-08T17:08:08.231 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=4457b0f1-028b-482b-8ee3-7ee05861e6cc --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-05-08T17:08:08.231 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-08T17:08:08.231 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-05-08T17:08:08.231 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-08T17:08:08.231 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/3' 2024-05-08T17:08:08.231 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/3/journal' 2024-05-08T17:08:08.231 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-05-08T17:08:08.231 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-05-08T17:08:08.231 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-05-08T17:08:08.231 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-05-08T17:08:08.231 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-08T17:08:08.231 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-08T17:08:08.231 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-08T17:08:08.231 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T17:08:08.232 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81110 2024-05-08T17:08:08.232 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.81110/$cluster-$name.asok' 2024-05-08T17:08:08.232 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.81110/$cluster-$name.asok' 2024-05-08T17:08:08.232 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-05-08T17:08:08.232 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-05-08T17:08:08.232 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-05-08T17:08:08.232 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-05-08T17:08:08.232 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-05-08T17:08:08.232 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-08T17:08:08.232 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-08T17:08:08.232 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-08T17:08:08.232 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-08T17:08:08.233 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-05-08T17:08:08.233 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-05-08T17:08:08.233 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/3 2024-05-08T17:08:08.233 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-05-08T17:08:08.233 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=20bef423-5778-4842-bbea-ef01f6743829 2024-05-08T17:08:08.234 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 20bef423-5778-4842-bbea-ef01f6743829' 2024-05-08T17:08:08.234 INFO:tasks.workunit.client.0.smithi172.stdout:add osd3 20bef423-5778-4842-bbea-ef01f6743829 2024-05-08T17:08:08.234 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-05-08T17:08:08.255 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQB4sTtmcn2TDxAAj3VMqaIe2WLShtCcQbO6GQ== 2024-05-08T17:08:08.255 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQB4sTtmcn2TDxAAj3VMqaIe2WLShtCcQbO6GQ=="}' 2024-05-08T17:08:08.255 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 20bef423-5778-4842-bbea-ef01f6743829 -i td/test-erasure-eio/3/new.json 2024-05-08T17:08:09.172 INFO:tasks.workunit.client.0.smithi172.stdout:3 2024-05-08T17:08:09.191 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/3/new.json 2024-05-08T17:08:09.192 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=4457b0f1-028b-482b-8ee3-7ee05861e6cc --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81110/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQB4sTtmcn2TDxAAj3VMqaIe2WLShtCcQbO6GQ== --osd-uuid 20bef423-5778-4842-bbea-ef01f6743829 2024-05-08T17:08:09.220 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:08:09.225+0000 7f5bd16347c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T17:08:09.292 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:08:09.297+0000 7f5bd16347c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T17:08:09.294 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:08:09.297+0000 7f5bd16347c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T17:08:09.294 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:08:09.297+0000 7f5bd16347c0 -1 bluestore(td/test-erasure-eio/3/block) _read_bdev_label failed to open td/test-erasure-eio/3/block: (2) No such file or directory 2024-05-08T17:08:09.295 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:08:09.297+0000 7f5bd16347c0 -1 bluestore(td/test-erasure-eio/3/block) _read_bdev_label failed to open td/test-erasure-eio/3/block: (2) No such file or directory 2024-05-08T17:08:09.295 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:08:09.297+0000 7f5bd16347c0 -1 bluestore(td/test-erasure-eio/3/block) _read_bdev_label failed to open td/test-erasure-eio/3/block: (2) No such file or directory 2024-05-08T17:08:09.473 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:08:09.477+0000 7f5bd16347c0 -1 bluestore(td/test-erasure-eio/3) _read_fsid unparsable uuid 2024-05-08T17:08:16.849 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/3/keyring 2024-05-08T17:08:16.849 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-05-08T17:08:16.851 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2024-05-08T17:08:16.851 INFO:tasks.workunit.client.0.smithi172.stdout:adding osd3 key to auth repository 2024-05-08T17:08:16.851 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-05-08T17:08:17.405 INFO:tasks.workunit.client.0.smithi172.stdout:start osd.3 2024-05-08T17:08:17.406 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2024-05-08T17:08:17.406 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-05-08T17:08:17.406 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-05-08T17:08:17.406 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=4457b0f1-028b-482b-8ee3-7ee05861e6cc --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81110/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-05-08T17:08:17.407 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-05-08T17:08:17.437 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:08:17.441+0000 7f8c7d0e27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T17:08:17.439 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:08:17.445+0000 7f8c7d0e27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T17:08:17.440 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:08:17.445+0000 7f8c7d0e27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T17:08:17.881 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2024-05-08T17:08:17.881 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-08T17:08:17.881 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-05-08T17:08:17.881 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-08T17:08:17.881 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-08T17:08:17.881 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:08:17.881 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-08T17:08:17.881 INFO:tasks.workunit.client.0.smithi172.stdout:0 2024-05-08T17:08:17.882 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:08:17.882 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T17:08:17.980 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:08:17.985+0000 7f8c7d0e27c0 -1 Falling back to public interface 2024-05-08T17:08:18.321 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:08:19.323 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:08:19.323 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:08:19.323 INFO:tasks.workunit.client.0.smithi172.stdout:1 2024-05-08T17:08:19.324 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-08T17:08:19.324 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:08:19.324 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T17:08:19.732 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:08:20.701 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:08:20.705+0000 7f8c7d0e27c0 -1 osd.3 0 log_to_monitors true 2024-05-08T17:08:20.734 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:08:20.734 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:08:20.734 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-08T17:08:20.734 INFO:tasks.workunit.client.0.smithi172.stdout:2 2024-05-08T17:08:20.734 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:08:20.735 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T17:08:21.488 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:08:22.490 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:08:22.490 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:08:22.491 INFO:tasks.workunit.client.0.smithi172.stdout:3 2024-05-08T17:08:22.491 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-05-08T17:08:22.491 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:08:22.491 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T17:08:23.145 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:08:24.147 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:08:24.147 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:08:24.147 INFO:tasks.workunit.client.0.smithi172.stdout:4 2024-05-08T17:08:24.148 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-05-08T17:08:24.148 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:08:24.148 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T17:08:24.662 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:08:25.664 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:08:25.664 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:08:25.664 INFO:tasks.workunit.client.0.smithi172.stdout:5 2024-05-08T17:08:25.665 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-05-08T17:08:25.665 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:08:25.665 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T17:08:26.323 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:08:27.324 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:08:27.325 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:08:27.325 INFO:tasks.workunit.client.0.smithi172.stdout:6 2024-05-08T17:08:27.325 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-05-08T17:08:27.325 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:08:27.326 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T17:08:27.970 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:08:28.972 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:08:28.972 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:08:28.972 INFO:tasks.workunit.client.0.smithi172.stdout:7 2024-05-08T17:08:28.973 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-05-08T17:08:28.973 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:08:28.973 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T17:08:29.604 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:08:30.606 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:08:30.606 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:08:30.607 INFO:tasks.workunit.client.0.smithi172.stdout:8 2024-05-08T17:08:30.607 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-05-08T17:08:30.607 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:08:30.607 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T17:08:31.109 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:08:32.110 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:08:32.111 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:08:32.111 INFO:tasks.workunit.client.0.smithi172.stdout:9 2024-05-08T17:08:32.111 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-05-08T17:08:32.111 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:08:32.112 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T17:08:32.596 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:08:33.597 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:08:33.598 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:08:33.598 INFO:tasks.workunit.client.0.smithi172.stdout:10 2024-05-08T17:08:33.598 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-05-08T17:08:33.598 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:08:33.598 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T17:08:34.019 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:08:35.020 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:08:35.020 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:08:35.020 INFO:tasks.workunit.client.0.smithi172.stdout:11 2024-05-08T17:08:35.021 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-05-08T17:08:35.021 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:08:35.021 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T17:08:35.557 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:08:36.558 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:08:36.559 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:08:36.559 INFO:tasks.workunit.client.0.smithi172.stdout:12 2024-05-08T17:08:36.559 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-05-08T17:08:36.559 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:08:36.560 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T17:08:37.542 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:08:38.544 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:08:38.544 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:08:38.544 INFO:tasks.workunit.client.0.smithi172.stdout:13 2024-05-08T17:08:38.545 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-05-08T17:08:38.545 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:08:38.545 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T17:08:39.390 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:08:40.392 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:08:40.392 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:08:40.392 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-05-08T17:08:40.392 INFO:tasks.workunit.client.0.smithi172.stdout:14 2024-05-08T17:08:40.393 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:08:40.393 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T17:08:41.020 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:08:42.022 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:08:42.022 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:08:42.022 INFO:tasks.workunit.client.0.smithi172.stdout:15 2024-05-08T17:08:42.023 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-05-08T17:08:42.023 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:08:42.023 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T17:08:43.209 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:08:44.211 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:08:44.229 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:08:44.229 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-05-08T17:08:44.230 INFO:tasks.workunit.client.0.smithi172.stdout:16 2024-05-08T17:08:44.230 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:08:44.230 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T17:08:44.822 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:08:45.824 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:08:45.824 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:08:45.825 INFO:tasks.workunit.client.0.smithi172.stdout:17 2024-05-08T17:08:45.825 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-05-08T17:08:45.826 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:08:45.826 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T17:08:46.779 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:08:47.781 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:08:47.781 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:08:47.781 INFO:tasks.workunit.client.0.smithi172.stdout:18 2024-05-08T17:08:47.782 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-05-08T17:08:47.782 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:08:47.782 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T17:08:48.399 INFO:tasks.workunit.client.0.smithi172.stdout:osd.3 up in weight 1 up_from 30 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/3929222293,v1:127.0.0.1:6827/3929222293] [v2:127.0.0.1:6828/3929222293,v1:127.0.0.1:6829/3929222293] exists,up 20bef423-5778-4842-bbea-ef01f6743829 2024-05-08T17:08:48.400 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-08T17:08:48.400 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-08T17:08:48.400 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-08T17:08:48.400 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: get_asok_path osd.0 2024-05-08T17:08:48.400 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-05-08T17:08:48.400 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-05-08T17:08:48.401 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-08T17:08:48.401 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T17:08:48.401 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81110 2024-05-08T17:08:48.401 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.81110/ceph-osd.0.asok 2024-05-08T17:08:48.401 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: CEPH_ARGS= 2024-05-08T17:08:48.401 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: ceph --admin-daemon /tmp/ceph-asok.81110/ceph-osd.0.asok log flush 2024-05-08T17:08:48.500 INFO:tasks.workunit.client.0.smithi172.stdout:{} 2024-05-08T17:08:48.511 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:56: setup_osds: grep 'load: jerasure.*lrc' td/test-erasure-eio/osd.0.log 2024-05-08T17:08:48.517 INFO:tasks.workunit.client.0.smithi172.stdout:2024-05-08T17:06:20.620+0000 7fb699b7f7c0 0 load: jerasure load: lrc 2024-05-08T17:08:48.518 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:258: TEST_rados_get_subread_eio_shard_0: local poolname=pool-jerasure 2024-05-08T17:08:48.518 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:259: TEST_rados_get_subread_eio_shard_0: create_erasure_coded_pool pool-jerasure 2 1 2024-05-08T17:08:48.518 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:67: create_erasure_coded_pool: local poolname=pool-jerasure 2024-05-08T17:08:48.518 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:68: create_erasure_coded_pool: shift 2024-05-08T17:08:48.518 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:69: create_erasure_coded_pool: local k=2 2024-05-08T17:08:48.518 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:70: create_erasure_coded_pool: shift 2024-05-08T17:08:48.519 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:71: create_erasure_coded_pool: local m=1 2024-05-08T17:08:48.519 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:72: create_erasure_coded_pool: shift 2024-05-08T17:08:48.519 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:74: create_erasure_coded_pool: ceph osd erasure-code-profile set myprofile plugin=jerasure k=2 m=1 crush-failure-domain=osd 2024-05-08T17:08:49.922 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:78: create_erasure_coded_pool: create_pool pool-jerasure 1 1 erasure myprofile 2024-05-08T17:08:49.923 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create pool-jerasure 1 1 erasure myprofile 2024-05-08T17:08:51.216 INFO:tasks.workunit.client.0.smithi172.stderr:pool 'pool-jerasure' already exists 2024-05-08T17:08:51.232 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-05-08T17:08:52.234 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:80: create_erasure_coded_pool: wait_for_clean 2024-05-08T17:08:52.235 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-05-08T17:08:52.235 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-05-08T17:08:52.235 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-05-08T17:08:52.235 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-05-08T17:08:52.236 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-05-08T17:08:52.236 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-05-08T17:08:52.237 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-05-08T17:08:52.237 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-05-08T17:08:52.237 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-05-08T17:08:52.366 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-05-08T17:08:52.366 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-05-08T17:08:52.366 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-05-08T17:08:52.366 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-05-08T17:08:52.366 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-05-08T17:08:52.366 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-05-08T17:08:52.828 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-05-08T17:08:52.829 INFO:tasks.workunit.client.0.smithi172.stderr:1 2024-05-08T17:08:52.829 INFO:tasks.workunit.client.0.smithi172.stderr:2 2024-05-08T17:08:52.829 INFO:tasks.workunit.client.0.smithi172.stderr:3' 2024-05-08T17:08:52.829 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-05-08T17:08:52.829 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T17:08:52.829 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-05-08T17:08:53.012 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=25769803808 2024-05-08T17:08:53.012 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 25769803808 2024-05-08T17:08:53.012 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803808' 2024-05-08T17:08:53.012 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T17:08:53.012 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-05-08T17:08:53.163 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=55834574876 2024-05-08T17:08:53.163 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 55834574876 2024-05-08T17:08:53.164 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803808 1-55834574876' 2024-05-08T17:08:53.164 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T17:08:53.164 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-05-08T17:08:53.318 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=85899345935 2024-05-08T17:08:53.318 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 85899345935 2024-05-08T17:08:53.318 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803808 1-55834574876 2-85899345935' 2024-05-08T17:08:53.318 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T17:08:53.318 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-05-08T17:08:53.471 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=128849018889 2024-05-08T17:08:53.471 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 128849018889 2024-05-08T17:08:53.471 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803808 1-55834574876 2-85899345935 3-128849018889' 2024-05-08T17:08:53.471 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-08T17:08:53.472 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-25769803808 2024-05-08T17:08:53.472 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-08T17:08:53.472 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-05-08T17:08:53.473 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-25769803808 2024-05-08T17:08:53.473 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-08T17:08:53.474 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=25769803808 2024-05-08T17:08:53.474 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 25769803808' 2024-05-08T17:08:53.475 INFO:tasks.workunit.client.0.smithi172.stdout:waiting osd.0 seq 25769803808 2024-05-08T17:08:53.475 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-05-08T17:08:53.897 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 25769803808 -lt 25769803808 2024-05-08T17:08:53.898 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-08T17:08:53.898 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-08T17:08:53.898 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-55834574876 2024-05-08T17:08:53.900 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-05-08T17:08:53.901 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-55834574876 2024-05-08T17:08:53.901 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-08T17:08:53.902 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=55834574876 2024-05-08T17:08:53.902 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 55834574876' 2024-05-08T17:08:53.903 INFO:tasks.workunit.client.0.smithi172.stdout:waiting osd.1 seq 55834574876 2024-05-08T17:08:53.903 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-05-08T17:08:54.343 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 55834574876 -lt 55834574876 2024-05-08T17:08:54.343 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-08T17:08:54.344 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-85899345935 2024-05-08T17:08:54.344 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-08T17:08:54.345 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-05-08T17:08:54.346 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-85899345935 2024-05-08T17:08:54.346 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-08T17:08:54.347 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=85899345935 2024-05-08T17:08:54.347 INFO:tasks.workunit.client.0.smithi172.stdout:waiting osd.2 seq 85899345935 2024-05-08T17:08:54.348 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 85899345935' 2024-05-08T17:08:54.348 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-05-08T17:08:54.792 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 85899345935 -lt 85899345935 2024-05-08T17:08:54.792 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-08T17:08:54.793 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-08T17:08:54.793 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-128849018889 2024-05-08T17:08:54.794 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-05-08T17:08:54.795 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-128849018889 2024-05-08T17:08:54.795 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-08T17:08:54.796 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=128849018889 2024-05-08T17:08:54.796 INFO:tasks.workunit.client.0.smithi172.stdout:waiting osd.3 seq 128849018889 2024-05-08T17:08:54.797 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 128849018889' 2024-05-08T17:08:54.797 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-05-08T17:08:55.207 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 128849018889 -lt 128849018889 2024-05-08T17:08:55.208 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-05-08T17:08:55.209 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-05-08T17:08:55.209 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-05-08T17:08:55.812 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-05-08T17:08:55.812 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-05-08T17:08:55.813 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-05-08T17:08:55.813 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-05-08T17:08:55.813 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-05-08T17:08:55.813 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-05-08T17:08:55.813 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-05-08T17:08:55.813 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-05-08T17:08:56.265 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-05-08T17:08:56.265 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-05-08T17:08:56.265 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-05-08T17:08:56.265 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-05-08T17:08:56.780 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-05-08T17:08:56.780 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-05-08T17:08:56.780 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-05-08T17:08:56.780 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:261: TEST_rados_get_subread_eio_shard_0: local shard_id=0 2024-05-08T17:08:56.780 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:262: TEST_rados_get_subread_eio_shard_0: rados_put_get_data eio td/test-erasure-eio 0 2024-05-08T17:08:56.780 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:147: rados_put_get_data: local inject=eio 2024-05-08T17:08:56.781 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:148: rados_put_get_data: shift 2024-05-08T17:08:56.781 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:149: rados_put_get_data: local dir=td/test-erasure-eio 2024-05-08T17:08:56.781 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:150: rados_put_get_data: shift 2024-05-08T17:08:56.781 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:151: rados_put_get_data: local shard_id=0 2024-05-08T17:08:56.781 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:152: rados_put_get_data: shift 2024-05-08T17:08:56.781 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:153: rados_put_get_data: local arg= 2024-05-08T17:08:56.781 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:157: rados_put_get_data: local poolname=pool-jerasure 2024-05-08T17:08:56.781 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:158: rados_put_get_data: local objname=obj-eio-81110-0 2024-05-08T17:08:56.781 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:159: rados_put_get_data: rados_put td/test-erasure-eio pool-jerasure obj-eio-81110-0 2024-05-08T17:08:56.781 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:90: rados_put: local dir=td/test-erasure-eio 2024-05-08T17:08:56.781 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:91: rados_put: local poolname=pool-jerasure 2024-05-08T17:08:56.781 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:92: rados_put: local objname=obj-eio-81110-0 2024-05-08T17:08:56.781 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-05-08T17:08:56.781 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 AAA 2024-05-08T17:08:56.782 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-05-08T17:08:56.782 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 BBB 2024-05-08T17:08:56.782 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-05-08T17:08:56.782 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 CCCC 2024-05-08T17:08:56.782 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-05-08T17:08:56.782 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 DDDD 2024-05-08T17:08:56.782 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:100: rados_put: rados --pool pool-jerasure put obj-eio-81110-0 td/test-erasure-eio/ORIGINAL 2024-05-08T17:08:57.103 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:160: rados_put_get_data: inject_eio ec data pool-jerasure obj-eio-81110-0 td/test-erasure-eio 0 2024-05-08T17:08:57.103 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2394: inject_eio: local pooltype=ec 2024-05-08T17:08:57.103 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2395: inject_eio: shift 2024-05-08T17:08:57.103 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2396: inject_eio: local which=data 2024-05-08T17:08:57.103 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2397: inject_eio: shift 2024-05-08T17:08:57.103 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2398: inject_eio: local poolname=pool-jerasure 2024-05-08T17:08:57.104 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2399: inject_eio: shift 2024-05-08T17:08:57.104 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2400: inject_eio: local objname=obj-eio-81110-0 2024-05-08T17:08:57.104 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2401: inject_eio: shift 2024-05-08T17:08:57.104 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2402: inject_eio: local dir=td/test-erasure-eio 2024-05-08T17:08:57.104 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2403: inject_eio: shift 2024-05-08T17:08:57.104 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2404: inject_eio: local shard_id=0 2024-05-08T17:08:57.104 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2405: inject_eio: shift 2024-05-08T17:08:57.104 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: get_osds pool-jerasure obj-eio-81110-0 2024-05-08T17:08:57.104 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2024-05-08T17:08:57.104 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-eio-81110-0 2024-05-08T17:08:57.105 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure obj-eio-81110-0 2024-05-08T17:08:57.105 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-05-08T17:08:57.547 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-05-08T17:08:57.547 INFO:tasks.workunit.client.0.smithi172.stderr:1 2024-05-08T17:08:57.547 INFO:tasks.workunit.client.0.smithi172.stderr:2' 2024-05-08T17:08:57.547 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 2 2024-05-08T17:08:57.547 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: initial_osds=('3' '1' '2') 2024-05-08T17:08:57.547 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: local -a initial_osds 2024-05-08T17:08:57.547 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2408: inject_eio: local osd_id=3 2024-05-08T17:08:57.547 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2409: inject_eio: '[' ec '!=' ec ']' 2024-05-08T17:08:57.548 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: cat td/test-erasure-eio/3/type 2024-05-08T17:08:57.548 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: type=bluestore 2024-05-08T17:08:57.548 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: set_config osd 3 bluestore_debug_inject_read_err true 2024-05-08T17:08:57.548 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-05-08T17:08:57.548 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-05-08T17:08:57.549 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-05-08T17:08:57.549 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-05-08T17:08:57.550 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-05-08T17:08:57.550 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-05-08T17:08:57.550 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-05-08T17:08:57.550 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-05-08T17:08:57.551 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-08T17:08:57.551 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T17:08:57.551 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81110 2024-05-08T17:08:57.551 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.81110/ceph-osd.3.asok 2024-05-08T17:08:57.551 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.81110/ceph-osd.3.asok config set bluestore_debug_inject_read_err true 2024-05-08T17:08:57.666 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-05-08T17:08:57.666 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local loop=0 2024-05-08T17:08:57.666 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: grep -q Invalid 2024-05-08T17:08:57.667 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: get_asok_path osd.3 2024-05-08T17:08:57.667 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-05-08T17:08:57.667 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-05-08T17:08:57.668 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-08T17:08:57.668 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T17:08:57.668 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81110 2024-05-08T17:08:57.668 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.81110/ceph-osd.3.asok 2024-05-08T17:08:57.669 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: CEPH_ARGS= 2024-05-08T17:08:57.669 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: ceph --admin-daemon /tmp/ceph-asok.81110/ceph-osd.3.asok injectdataerr pool-jerasure obj-eio-81110-0 0 2024-05-08T17:08:57.795 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:161: rados_put_get_data: rados_get td/test-erasure-eio pool-jerasure obj-eio-81110-0 2024-05-08T17:08:57.795 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2024-05-08T17:08:57.795 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2024-05-08T17:08:57.795 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=obj-eio-81110-0 2024-05-08T17:08:57.795 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=ok 2024-05-08T17:08:57.795 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' ok = fail ']' 2024-05-08T17:08:57.795 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:120: rados_get: rados --pool pool-jerasure get obj-eio-81110-0 td/test-erasure-eio/COPY 2024-05-08T17:08:57.833 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:121: rados_get: diff td/test-erasure-eio/ORIGINAL td/test-erasure-eio/COPY 2024-05-08T17:08:57.834 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:122: rados_get: rm td/test-erasure-eio/COPY 2024-05-08T17:08:57.834 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:163: rados_put_get_data: '[' '' = recovery ']' 2024-05-08T17:08:57.835 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:181: rados_put_get_data: expr 0 + 1 2024-05-08T17:08:57.837 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:181: rados_put_get_data: shard_id=1 2024-05-08T17:08:57.837 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:182: rados_put_get_data: inject_eio ec data pool-jerasure obj-eio-81110-0 td/test-erasure-eio 1 2024-05-08T17:08:57.837 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2394: inject_eio: local pooltype=ec 2024-05-08T17:08:57.837 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2395: inject_eio: shift 2024-05-08T17:08:57.837 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2396: inject_eio: local which=data 2024-05-08T17:08:57.837 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2397: inject_eio: shift 2024-05-08T17:08:57.837 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2398: inject_eio: local poolname=pool-jerasure 2024-05-08T17:08:57.837 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2399: inject_eio: shift 2024-05-08T17:08:57.838 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2400: inject_eio: local objname=obj-eio-81110-0 2024-05-08T17:08:57.838 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2401: inject_eio: shift 2024-05-08T17:08:57.838 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2402: inject_eio: local dir=td/test-erasure-eio 2024-05-08T17:08:57.838 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2403: inject_eio: shift 2024-05-08T17:08:57.838 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2404: inject_eio: local shard_id=1 2024-05-08T17:08:57.838 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2405: inject_eio: shift 2024-05-08T17:08:57.838 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: get_osds pool-jerasure obj-eio-81110-0 2024-05-08T17:08:57.838 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2024-05-08T17:08:57.838 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-eio-81110-0 2024-05-08T17:08:57.838 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure obj-eio-81110-0 2024-05-08T17:08:57.838 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-05-08T17:08:58.233 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-05-08T17:08:58.233 INFO:tasks.workunit.client.0.smithi172.stderr:1 2024-05-08T17:08:58.234 INFO:tasks.workunit.client.0.smithi172.stderr:2' 2024-05-08T17:08:58.234 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 2 2024-05-08T17:08:58.234 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: initial_osds=('3' '1' '2') 2024-05-08T17:08:58.234 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: local -a initial_osds 2024-05-08T17:08:58.234 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2408: inject_eio: local osd_id=1 2024-05-08T17:08:58.234 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2409: inject_eio: '[' ec '!=' ec ']' 2024-05-08T17:08:58.235 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: cat td/test-erasure-eio/1/type 2024-05-08T17:08:58.235 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: type=bluestore 2024-05-08T17:08:58.235 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2024-05-08T17:08:58.235 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-05-08T17:08:58.235 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=1 2024-05-08T17:08:58.235 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-05-08T17:08:58.235 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-05-08T17:08:58.236 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-05-08T17:08:58.237 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.1 2024-05-08T17:08:58.237 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-05-08T17:08:58.237 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-05-08T17:08:58.237 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-08T17:08:58.237 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T17:08:58.237 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81110 2024-05-08T17:08:58.237 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.81110/ceph-osd.1.asok 2024-05-08T17:08:58.237 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.81110/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2024-05-08T17:08:58.354 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-05-08T17:08:58.354 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local loop=0 2024-05-08T17:08:58.355 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: grep -q Invalid 2024-05-08T17:08:58.355 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: get_asok_path osd.1 2024-05-08T17:08:58.355 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-05-08T17:08:58.356 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-05-08T17:08:58.356 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-08T17:08:58.357 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T17:08:58.357 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81110 2024-05-08T17:08:58.357 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.81110/ceph-osd.1.asok 2024-05-08T17:08:58.357 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: CEPH_ARGS= 2024-05-08T17:08:58.357 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: ceph --admin-daemon /tmp/ceph-asok.81110/ceph-osd.1.asok injectdataerr pool-jerasure obj-eio-81110-0 1 2024-05-08T17:08:58.466 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:183: rados_put_get_data: rados_get td/test-erasure-eio pool-jerasure obj-eio-81110-0 fail 2024-05-08T17:08:58.466 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2024-05-08T17:08:58.466 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2024-05-08T17:08:58.466 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=obj-eio-81110-0 2024-05-08T17:08:58.466 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=fail 2024-05-08T17:08:58.466 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' fail = fail ']' 2024-05-08T17:08:58.467 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:114: rados_get: rados --pool pool-jerasure get obj-eio-81110-0 td/test-erasure-eio/COPY 2024-05-08T17:08:58.500 INFO:tasks.workunit.client.0.smithi172.stderr:error getting pool-jerasure/obj-eio-81110-0: (5) Input/output error 2024-05-08T17:08:58.502 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:115: rados_get: return 2024-05-08T17:08:58.502 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:184: rados_put_get_data: rm td/test-erasure-eio/ORIGINAL 2024-05-08T17:08:58.503 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:263: TEST_rados_get_subread_eio_shard_0: delete_erasure_coded_pool pool-jerasure 2024-05-08T17:08:58.503 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:84: delete_erasure_coded_pool: local poolname=pool-jerasure 2024-05-08T17:08:58.503 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:85: delete_erasure_coded_pool: ceph osd pool delete pool-jerasure pool-jerasure --yes-i-really-really-mean-it 2024-05-08T17:08:59.036 INFO:tasks.workunit.client.0.smithi172.stderr:pool 'pool-jerasure' does not exist 2024-05-08T17:08:59.051 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:86: delete_erasure_coded_pool: ceph osd erasure-code-profile rm myprofile 2024-05-08T17:08:59.783 INFO:tasks.workunit.client.0.smithi172.stderr:erasure-code-profile myprofile does not exist 2024-05-08T17:08:59.800 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:42: run: teardown td/test-erasure-eio 2024-05-08T17:08:59.800 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-eio 2024-05-08T17:08:59.800 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-05-08T17:08:59.800 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-eio KILL 2024-05-08T17:08:59.801 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-08T17:08:59.801 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-08T17:08:59.801 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-08T17:08:59.801 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-08T17:08:59.801 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-08T17:09:00.114 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-08T17:09:00.114 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-05-08T17:09:00.115 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-05-08T17:09:00.116 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-05-08T17:09:00.117 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-05-08T17:09:00.117 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-05-08T17:09:00.118 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-05-08T17:09:00.119 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-08T17:09:00.119 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-05-08T17:09:00.119 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-05-08T17:09:00.119 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-08T17:09:00.120 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-05-08T17:09:00.121 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-05-08T17:09:00.122 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-eio 2024-05-08T17:09:00.145 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-05-08T17:09:00.145 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T17:09:00.145 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81110 2024-05-08T17:09:00.145 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.81110 2024-05-08T17:09:00.147 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-05-08T17:09:00.147 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-05-08T17:09:00.147 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:32: run: for func in $funcs 2024-05-08T17:09:00.147 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:33: run: setup td/test-erasure-eio 2024-05-08T17:09:00.147 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/test-erasure-eio 2024-05-08T17:09:00.147 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/test-erasure-eio 2024-05-08T17:09:00.147 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-eio 2024-05-08T17:09:00.147 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-05-08T17:09:00.148 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-eio KILL 2024-05-08T17:09:00.148 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-08T17:09:00.148 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-08T17:09:00.148 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-08T17:09:00.148 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-08T17:09:00.148 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-08T17:09:00.149 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-08T17:09:00.149 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-05-08T17:09:00.150 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-05-08T17:09:00.150 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-05-08T17:09:00.151 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-05-08T17:09:00.152 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-05-08T17:09:00.152 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-05-08T17:09:00.153 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-08T17:09:00.153 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-05-08T17:09:00.154 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-05-08T17:09:00.154 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-08T17:09:00.154 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-05-08T17:09:00.156 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-05-08T17:09:00.156 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-eio 2024-05-08T17:09:00.157 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-05-08T17:09:00.157 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T17:09:00.158 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81110 2024-05-08T17:09:00.158 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.81110 2024-05-08T17:09:00.159 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-05-08T17:09:00.159 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-05-08T17:09:00.159 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/test-erasure-eio 2024-05-08T17:09:00.160 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-05-08T17:09:00.160 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T17:09:00.160 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81110 2024-05-08T17:09:00.160 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.81110 2024-05-08T17:09:00.162 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-05-08T17:09:00.162 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-05-08T17:09:00.162 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-05-08T17:09:00.162 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/test-erasure-eio 1' TERM HUP INT 2024-05-08T17:09:00.163 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:34: run: run_mon td/test-erasure-eio a 2024-05-08T17:09:00.163 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/test-erasure-eio 2024-05-08T17:09:00.163 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-05-08T17:09:00.163 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-05-08T17:09:00.163 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-05-08T17:09:00.163 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/test-erasure-eio/a 2024-05-08T17:09:00.163 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-eio/a --run-dir=td/test-erasure-eio 2024-05-08T17:09:00.591 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-05-08T17:09:00.591 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-08T17:09:00.591 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-08T17:09:00.592 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-08T17:09:00.592 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T17:09:00.592 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81110 2024-05-08T17:09:00.592 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.81110/$cluster-$name.asok' 2024-05-08T17:09:00.592 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-eio/a '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.81110/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-eio/log --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-05-08T17:09:00.827 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-05-08T17:09:00.829 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-05-08T17:09:00.829 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-05-08T17:09:00.829 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-05-08T17:09:00.829 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-05-08T17:09:00.829 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-05-08T17:09:00.830 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-05-08T17:09:00.830 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-05-08T17:09:00.830 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-05-08T17:09:00.831 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-08T17:09:00.831 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T17:09:00.831 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81110 2024-05-08T17:09:00.831 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.81110/ceph-mon.a.asok 2024-05-08T17:09:00.832 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-05-08T17:09:00.832 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.81110/ceph-mon.a.asok config get fsid 2024-05-08T17:09:00.943 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-05-08T17:09:00.943 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-05-08T17:09:00.943 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-05-08T17:09:00.943 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-05-08T17:09:00.946 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-05-08T17:09:00.946 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-05-08T17:09:00.946 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-05-08T17:09:00.946 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-05-08T17:09:00.946 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-08T17:09:00.946 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T17:09:00.947 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81110 2024-05-08T17:09:00.947 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.81110/ceph-mon.a.asok 2024-05-08T17:09:00.947 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-05-08T17:09:00.947 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.81110/ceph-mon.a.asok config get mon_host 2024-05-08T17:09:01.062 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:35: run: run_mgr td/test-erasure-eio x 2024-05-08T17:09:01.062 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/test-erasure-eio 2024-05-08T17:09:01.062 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-05-08T17:09:01.063 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-05-08T17:09:01.063 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-05-08T17:09:01.063 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/test-erasure-eio/x 2024-05-08T17:09:01.063 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-05-08T17:09:01.982 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-05-08T17:09:01.982 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-08T17:09:01.982 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-08T17:09:01.982 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-08T17:09:01.982 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T17:09:01.982 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81110 2024-05-08T17:09:01.983 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.81110/$cluster-$name.asok' 2024-05-08T17:09:01.983 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-05-08T17:09:01.984 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-eio/x '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.81110/$cluster-$name.asok' --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-05-08T17:09:02.022 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:36: run: create_pool rbd 4 2024-05-08T17:09:02.022 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-05-08T17:09:02.347 INFO:tasks.workunit.client.0.smithi172.stderr:pool 'rbd' already exists 2024-05-08T17:09:02.364 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-05-08T17:09:03.367 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: get_asok_path mon.a 2024-05-08T17:09:03.367 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-05-08T17:09:03.367 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-05-08T17:09:03.368 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-08T17:09:03.368 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T17:09:03.368 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81110 2024-05-08T17:09:03.368 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.81110/ceph-mon.a.asok 2024-05-08T17:09:03.368 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: CEPH_ARGS= 2024-05-08T17:09:03.368 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: ceph --admin-daemon /tmp/ceph-asok.81110/ceph-mon.a.asok log flush 2024-05-08T17:09:03.465 INFO:tasks.workunit.client.0.smithi172.stdout:{} 2024-05-08T17:09:03.477 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:40: run: grep 'load: jerasure.*lrc' td/test-erasure-eio/mon.a.log 2024-05-08T17:09:03.479 INFO:tasks.workunit.client.0.smithi172.stdout:2024-05-08T17:09:00.640+0000 7f7034bacc80 0 load: jerasure load: lrc 2024-05-08T17:09:03.480 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:41: run: TEST_rados_get_subread_eio_shard_1 td/test-erasure-eio 2024-05-08T17:09:03.480 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:267: TEST_rados_get_subread_eio_shard_1: local dir=td/test-erasure-eio 2024-05-08T17:09:03.481 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:268: TEST_rados_get_subread_eio_shard_1: setup_osds 4 2024-05-08T17:09:03.481 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:47: setup_osds: local count=4 2024-05-08T17:09:03.481 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:48: setup_osds: shift 2024-05-08T17:09:03.481 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: expr 4 - 1 2024-05-08T17:09:03.490 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: seq 0 3 2024-05-08T17:09:03.490 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-05-08T17:09:03.490 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 0 2024-05-08T17:09:03.490 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-05-08T17:09:03.490 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-05-08T17:09:03.491 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-05-08T17:09:03.491 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-05-08T17:09:03.491 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/0 2024-05-08T17:09:03.491 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=4457b0f1-028b-482b-8ee3-7ee05861e6cc --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-05-08T17:09:03.491 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-08T17:09:03.492 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-05-08T17:09:03.492 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-08T17:09:03.492 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/0' 2024-05-08T17:09:03.492 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/0/journal' 2024-05-08T17:09:03.492 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-05-08T17:09:03.492 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-05-08T17:09:03.492 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-05-08T17:09:03.493 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-05-08T17:09:03.493 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-08T17:09:03.493 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-08T17:09:03.493 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-08T17:09:03.493 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T17:09:03.493 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81110 2024-05-08T17:09:03.494 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.81110/$cluster-$name.asok' 2024-05-08T17:09:03.494 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.81110/$cluster-$name.asok' 2024-05-08T17:09:03.494 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-05-08T17:09:03.494 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-05-08T17:09:03.494 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-05-08T17:09:03.494 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-05-08T17:09:03.494 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-05-08T17:09:03.495 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-08T17:09:03.495 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-08T17:09:03.495 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-08T17:09:03.495 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-08T17:09:03.495 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-05-08T17:09:03.495 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-05-08T17:09:03.495 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/0 2024-05-08T17:09:03.495 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-05-08T17:09:03.496 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=55a1faff-314e-4591-84d2-7b11a88c50b1 2024-05-08T17:09:03.496 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 55a1faff-314e-4591-84d2-7b11a88c50b1' 2024-05-08T17:09:03.496 INFO:tasks.workunit.client.0.smithi172.stdout:add osd0 55a1faff-314e-4591-84d2-7b11a88c50b1 2024-05-08T17:09:03.497 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-05-08T17:09:03.509 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCvsTtmYi/DHhAAa9PfNLSveMXFyyztbtc3Rg== 2024-05-08T17:09:03.510 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCvsTtmYi/DHhAAa9PfNLSveMXFyyztbtc3Rg=="}' 2024-05-08T17:09:03.510 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 55a1faff-314e-4591-84d2-7b11a88c50b1 -i td/test-erasure-eio/0/new.json 2024-05-08T17:09:03.823 INFO:tasks.workunit.client.0.smithi172.stdout:0 2024-05-08T17:09:03.838 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/0/new.json 2024-05-08T17:09:03.839 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=4457b0f1-028b-482b-8ee3-7ee05861e6cc --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81110/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCvsTtmYi/DHhAAa9PfNLSveMXFyyztbtc3Rg== --osd-uuid 55a1faff-314e-4591-84d2-7b11a88c50b1 2024-05-08T17:09:03.868 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:09:03.872+0000 7f070f27f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T17:09:03.891 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:09:03.896+0000 7f070f27f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T17:09:03.893 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:09:03.896+0000 7f070f27f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T17:09:03.893 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:09:03.896+0000 7f070f27f7c0 -1 bluestore(td/test-erasure-eio/0/block) _read_bdev_label failed to open td/test-erasure-eio/0/block: (2) No such file or directory 2024-05-08T17:09:03.893 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:09:03.896+0000 7f070f27f7c0 -1 bluestore(td/test-erasure-eio/0/block) _read_bdev_label failed to open td/test-erasure-eio/0/block: (2) No such file or directory 2024-05-08T17:09:03.893 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:09:03.896+0000 7f070f27f7c0 -1 bluestore(td/test-erasure-eio/0/block) _read_bdev_label failed to open td/test-erasure-eio/0/block: (2) No such file or directory 2024-05-08T17:09:03.923 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:09:03.924+0000 7f070f27f7c0 -1 bluestore(td/test-erasure-eio/0) _read_fsid unparsable uuid 2024-05-08T17:09:10.716 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/0/keyring 2024-05-08T17:09:10.716 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-05-08T17:09:10.717 INFO:tasks.workunit.client.0.smithi172.stdout:adding osd0 key to auth repository 2024-05-08T17:09:10.718 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-05-08T17:09:10.718 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-05-08T17:09:11.265 INFO:tasks.workunit.client.0.smithi172.stdout:start osd.0 2024-05-08T17:09:11.265 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-05-08T17:09:11.266 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=4457b0f1-028b-482b-8ee3-7ee05861e6cc --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81110/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-05-08T17:09:11.266 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-05-08T17:09:11.266 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-05-08T17:09:11.266 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-05-08T17:09:11.296 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:09:11.300+0000 7f7aef3647c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T17:09:11.298 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:09:11.300+0000 7f7aef3647c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T17:09:11.299 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:09:11.304+0000 7f7aef3647c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T17:09:11.662 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-05-08T17:09:11.663 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-08T17:09:11.663 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-05-08T17:09:11.663 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-08T17:09:11.663 INFO:tasks.workunit.client.0.smithi172.stdout:0 2024-05-08T17:09:11.664 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-08T17:09:11.664 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:09:11.664 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-08T17:09:11.664 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:09:11.664 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T17:09:11.865 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:09:11.868+0000 7f7aef3647c0 -1 Falling back to public interface 2024-05-08T17:09:12.090 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:09:13.092 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:09:13.092 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:09:13.093 INFO:tasks.workunit.client.0.smithi172.stdout:1 2024-05-08T17:09:13.093 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-08T17:09:13.094 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:09:13.094 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T17:09:13.500 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:09:14.474 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:09:14.472+0000 7f7aef3647c0 -1 osd.0 0 log_to_monitors true 2024-05-08T17:09:14.501 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:09:14.502 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:09:14.502 INFO:tasks.workunit.client.0.smithi172.stdout:2 2024-05-08T17:09:14.502 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-08T17:09:14.502 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:09:14.503 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T17:09:15.180 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:09:16.182 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:09:16.182 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:09:16.182 INFO:tasks.workunit.client.0.smithi172.stdout:3 2024-05-08T17:09:16.183 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-05-08T17:09:16.183 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:09:16.183 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T17:09:16.631 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:09:17.633 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:09:17.633 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:09:17.633 INFO:tasks.workunit.client.0.smithi172.stdout:4 2024-05-08T17:09:17.634 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-05-08T17:09:17.634 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:09:17.634 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T17:09:18.106 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:09:19.107 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:09:19.107 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:09:19.107 INFO:tasks.workunit.client.0.smithi172.stdout:5 2024-05-08T17:09:19.108 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-05-08T17:09:19.108 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:09:19.108 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T17:09:19.640 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:09:20.642 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:09:20.642 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:09:20.642 INFO:tasks.workunit.client.0.smithi172.stdout:6 2024-05-08T17:09:20.643 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-05-08T17:09:20.643 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:09:20.643 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T17:09:21.102 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:09:22.104 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:09:22.104 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:09:22.104 INFO:tasks.workunit.client.0.smithi172.stdout:7 2024-05-08T17:09:22.104 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-05-08T17:09:22.105 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:09:22.105 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T17:09:22.616 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:09:23.618 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:09:23.619 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:09:23.619 INFO:tasks.workunit.client.0.smithi172.stdout:8 2024-05-08T17:09:23.619 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-05-08T17:09:23.619 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:09:23.620 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T17:09:24.089 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:09:25.091 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:09:25.091 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:09:25.091 INFO:tasks.workunit.client.0.smithi172.stdout:9 2024-05-08T17:09:25.092 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-05-08T17:09:25.092 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:09:25.092 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T17:09:25.644 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:09:26.645 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:09:26.645 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:09:26.646 INFO:tasks.workunit.client.0.smithi172.stdout:10 2024-05-08T17:09:26.646 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-05-08T17:09:26.646 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:09:26.646 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T17:09:27.198 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:09:28.197 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:09:28.197 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:09:28.197 INFO:tasks.workunit.client.0.smithi172.stdout:11 2024-05-08T17:09:28.198 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-05-08T17:09:28.198 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:09:28.198 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T17:09:28.950 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:09:29.951 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:09:29.952 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:09:29.952 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-05-08T17:09:29.952 INFO:tasks.workunit.client.0.smithi172.stdout:12 2024-05-08T17:09:29.952 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:09:29.953 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T17:09:30.753 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:09:31.754 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:09:31.755 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:09:31.755 INFO:tasks.workunit.client.0.smithi172.stdout:13 2024-05-08T17:09:31.755 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-05-08T17:09:31.755 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:09:31.756 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T17:09:32.590 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:09:33.591 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:09:33.591 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:09:33.591 INFO:tasks.workunit.client.0.smithi172.stdout:14 2024-05-08T17:09:33.592 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-05-08T17:09:33.592 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:09:33.592 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T17:09:34.139 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:09:35.140 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:09:35.141 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:09:35.141 INFO:tasks.workunit.client.0.smithi172.stdout:15 2024-05-08T17:09:35.141 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-05-08T17:09:35.141 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:09:35.141 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T17:09:36.060 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:09:37.061 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:09:37.061 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:09:37.061 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-05-08T17:09:37.062 INFO:tasks.workunit.client.0.smithi172.stdout:16 2024-05-08T17:09:37.062 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:09:37.062 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T17:09:38.253 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:09:39.255 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:09:39.256 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:09:39.256 INFO:tasks.workunit.client.0.smithi172.stdout:17 2024-05-08T17:09:39.256 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-05-08T17:09:39.257 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:09:39.257 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T17:09:39.764 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:09:40.765 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:09:40.765 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:09:40.766 INFO:tasks.workunit.client.0.smithi172.stdout:18 2024-05-08T17:09:40.766 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-05-08T17:09:40.766 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:09:40.766 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T17:09:41.211 INFO:tasks.workunit.client.0.smithi172.stdout:osd.0 up in weight 1 up_from 6 up_thru 6 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3776253806,v1:127.0.0.1:6803/3776253806] [v2:127.0.0.1:6804/3776253806,v1:127.0.0.1:6805/3776253806] exists,up 55a1faff-314e-4591-84d2-7b11a88c50b1 2024-05-08T17:09:41.214 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-08T17:09:41.214 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-08T17:09:41.214 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-08T17:09:41.214 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-05-08T17:09:41.214 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 1 2024-05-08T17:09:41.214 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-05-08T17:09:41.214 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-05-08T17:09:41.214 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-05-08T17:09:41.214 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-05-08T17:09:41.215 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/1 2024-05-08T17:09:41.215 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=4457b0f1-028b-482b-8ee3-7ee05861e6cc --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-05-08T17:09:41.215 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-08T17:09:41.215 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-05-08T17:09:41.215 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-08T17:09:41.215 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/1' 2024-05-08T17:09:41.215 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/1/journal' 2024-05-08T17:09:41.215 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-05-08T17:09:41.215 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-05-08T17:09:41.215 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-05-08T17:09:41.215 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-05-08T17:09:41.215 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-08T17:09:41.215 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-08T17:09:41.215 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-08T17:09:41.215 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T17:09:41.216 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81110 2024-05-08T17:09:41.216 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.81110/$cluster-$name.asok' 2024-05-08T17:09:41.216 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.81110/$cluster-$name.asok' 2024-05-08T17:09:41.216 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-05-08T17:09:41.216 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-05-08T17:09:41.216 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-05-08T17:09:41.216 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-05-08T17:09:41.216 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-05-08T17:09:41.216 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-08T17:09:41.216 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-08T17:09:41.216 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-08T17:09:41.216 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-08T17:09:41.216 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-05-08T17:09:41.216 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-05-08T17:09:41.216 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/1 2024-05-08T17:09:41.217 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-05-08T17:09:41.217 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=1121535a-ad1b-4120-937a-67da12487058 2024-05-08T17:09:41.218 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 1121535a-ad1b-4120-937a-67da12487058' 2024-05-08T17:09:41.218 INFO:tasks.workunit.client.0.smithi172.stdout:add osd1 1121535a-ad1b-4120-937a-67da12487058 2024-05-08T17:09:41.218 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-05-08T17:09:41.232 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDVsTtmwJ4+DhAAU7tco80NVQfNgxGY6SuWhQ== 2024-05-08T17:09:41.232 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDVsTtmwJ4+DhAAU7tco80NVQfNgxGY6SuWhQ=="}' 2024-05-08T17:09:41.232 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 1121535a-ad1b-4120-937a-67da12487058 -i td/test-erasure-eio/1/new.json 2024-05-08T17:09:41.696 INFO:tasks.workunit.client.0.smithi172.stdout:1 2024-05-08T17:09:41.711 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/1/new.json 2024-05-08T17:09:41.712 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=4457b0f1-028b-482b-8ee3-7ee05861e6cc --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81110/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDVsTtmwJ4+DhAAU7tco80NVQfNgxGY6SuWhQ== --osd-uuid 1121535a-ad1b-4120-937a-67da12487058 2024-05-08T17:09:41.746 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:09:41.751+0000 7f9bee4577c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T17:09:41.775 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:09:41.779+0000 7f9bee4577c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T17:09:41.776 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:09:41.783+0000 7f9bee4577c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T17:09:41.777 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:09:41.783+0000 7f9bee4577c0 -1 bluestore(td/test-erasure-eio/1/block) _read_bdev_label failed to open td/test-erasure-eio/1/block: (2) No such file or directory 2024-05-08T17:09:41.777 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:09:41.783+0000 7f9bee4577c0 -1 bluestore(td/test-erasure-eio/1/block) _read_bdev_label failed to open td/test-erasure-eio/1/block: (2) No such file or directory 2024-05-08T17:09:41.777 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:09:41.783+0000 7f9bee4577c0 -1 bluestore(td/test-erasure-eio/1/block) _read_bdev_label failed to open td/test-erasure-eio/1/block: (2) No such file or directory 2024-05-08T17:09:41.806 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:09:41.811+0000 7f9bee4577c0 -1 bluestore(td/test-erasure-eio/1) _read_fsid unparsable uuid 2024-05-08T17:09:48.735 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/1/keyring 2024-05-08T17:09:48.735 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-05-08T17:09:48.736 INFO:tasks.workunit.client.0.smithi172.stdout:adding osd1 key to auth repository 2024-05-08T17:09:48.737 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-05-08T17:09:48.737 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-05-08T17:09:49.311 INFO:tasks.workunit.client.0.smithi172.stdout:start osd.1 2024-05-08T17:09:49.312 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-05-08T17:09:49.312 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=4457b0f1-028b-482b-8ee3-7ee05861e6cc --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81110/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-05-08T17:09:49.312 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-05-08T17:09:49.312 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-05-08T17:09:49.312 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-05-08T17:09:49.343 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:09:49.347+0000 7f68c92367c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T17:09:49.372 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:09:49.379+0000 7f68c92367c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T17:09:49.374 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:09:49.379+0000 7f68c92367c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T17:09:49.717 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-05-08T17:09:49.717 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-08T17:09:49.717 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-05-08T17:09:49.717 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-08T17:09:49.717 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-08T17:09:49.717 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:09:49.717 INFO:tasks.workunit.client.0.smithi172.stdout:0 2024-05-08T17:09:49.718 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-08T17:09:49.718 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:09:49.718 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T17:09:49.934 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:09:49.939+0000 7f68c92367c0 -1 Falling back to public interface 2024-05-08T17:09:50.151 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:09:51.153 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:09:51.154 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:09:51.154 INFO:tasks.workunit.client.0.smithi172.stdout:1 2024-05-08T17:09:51.155 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-08T17:09:51.155 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:09:51.156 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T17:09:51.567 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:09:52.535 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:09:52.539+0000 7f68c92367c0 -1 osd.1 0 log_to_monitors true 2024-05-08T17:09:52.568 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:09:52.569 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:09:52.569 INFO:tasks.workunit.client.0.smithi172.stdout:2 2024-05-08T17:09:52.569 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-08T17:09:52.569 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T17:09:52.569 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:09:53.479 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:09:54.481 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:09:54.481 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:09:54.481 INFO:tasks.workunit.client.0.smithi172.stdout:3 2024-05-08T17:09:54.482 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-05-08T17:09:54.482 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:09:54.482 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T17:09:54.937 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:09:55.939 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:09:55.939 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:09:55.939 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-05-08T17:09:55.939 INFO:tasks.workunit.client.0.smithi172.stdout:4 2024-05-08T17:09:55.939 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:09:55.940 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T17:09:56.387 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:09:57.389 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:09:57.389 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:09:57.390 INFO:tasks.workunit.client.0.smithi172.stdout:5 2024-05-08T17:09:57.390 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-05-08T17:09:57.391 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:09:57.391 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T17:09:57.949 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:09:58.951 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:09:58.951 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:09:58.951 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-05-08T17:09:58.951 INFO:tasks.workunit.client.0.smithi172.stdout:6 2024-05-08T17:09:58.952 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:09:58.952 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T17:09:59.467 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:10:00.469 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:10:00.469 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:10:00.469 INFO:tasks.workunit.client.0.smithi172.stdout:7 2024-05-08T17:10:00.469 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-05-08T17:10:00.470 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:10:00.470 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T17:10:00.942 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:10:01.944 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:10:01.944 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:10:01.944 INFO:tasks.workunit.client.0.smithi172.stdout:8 2024-05-08T17:10:01.945 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-05-08T17:10:01.945 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:10:01.945 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T17:10:02.408 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:10:03.410 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:10:03.410 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:10:03.410 INFO:tasks.workunit.client.0.smithi172.stdout:9 2024-05-08T17:10:03.411 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-05-08T17:10:03.411 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:10:03.411 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T17:10:03.932 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:10:04.934 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:10:04.934 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:10:04.934 INFO:tasks.workunit.client.0.smithi172.stdout:10 2024-05-08T17:10:04.935 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-05-08T17:10:04.935 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:10:04.935 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T17:10:05.437 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:10:06.439 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:10:06.439 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:10:06.439 INFO:tasks.workunit.client.0.smithi172.stdout:11 2024-05-08T17:10:06.440 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-05-08T17:10:06.440 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:10:06.440 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T17:10:06.931 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:10:07.933 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:10:07.933 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:10:07.933 INFO:tasks.workunit.client.0.smithi172.stdout:12 2024-05-08T17:10:07.933 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-05-08T17:10:07.934 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:10:07.934 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T17:10:08.467 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:10:09.469 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:10:09.469 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:10:09.469 INFO:tasks.workunit.client.0.smithi172.stdout:13 2024-05-08T17:10:09.470 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-05-08T17:10:09.470 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:10:09.470 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T17:10:10.581 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:10:11.583 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:10:11.583 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:10:11.583 INFO:tasks.workunit.client.0.smithi172.stdout:14 2024-05-08T17:10:11.584 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-05-08T17:10:11.584 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:10:11.584 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T17:10:12.471 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:10:13.473 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:10:13.473 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:10:13.473 INFO:tasks.workunit.client.0.smithi172.stdout:15 2024-05-08T17:10:13.474 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-05-08T17:10:13.474 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:10:13.474 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T17:10:14.465 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:10:15.467 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:10:15.467 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:10:15.467 INFO:tasks.workunit.client.0.smithi172.stdout:16 2024-05-08T17:10:15.468 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-05-08T17:10:15.468 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:10:15.468 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T17:10:16.800 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:10:17.801 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:10:17.801 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:10:17.802 INFO:tasks.workunit.client.0.smithi172.stdout:17 2024-05-08T17:10:17.802 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-05-08T17:10:17.802 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:10:17.802 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T17:10:18.757 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:10:19.759 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:10:19.760 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:10:19.760 INFO:tasks.workunit.client.0.smithi172.stdout:18 2024-05-08T17:10:19.761 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-05-08T17:10:19.761 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:10:19.761 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T17:10:20.281 INFO:tasks.workunit.client.0.smithi172.stdout:osd.1 up in weight 1 up_from 13 up_thru 13 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/486434952,v1:127.0.0.1:6811/486434952] [v2:127.0.0.1:6812/486434952,v1:127.0.0.1:6813/486434952] exists,up 1121535a-ad1b-4120-937a-67da12487058 2024-05-08T17:10:20.283 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-08T17:10:20.283 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-08T17:10:20.283 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-08T17:10:20.283 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-05-08T17:10:20.284 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 2 2024-05-08T17:10:20.284 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-05-08T17:10:20.284 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-05-08T17:10:20.284 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-05-08T17:10:20.284 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-05-08T17:10:20.284 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/2 2024-05-08T17:10:20.284 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=4457b0f1-028b-482b-8ee3-7ee05861e6cc --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-05-08T17:10:20.284 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-08T17:10:20.284 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-05-08T17:10:20.284 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-08T17:10:20.284 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/2' 2024-05-08T17:10:20.284 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/2/journal' 2024-05-08T17:10:20.284 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-05-08T17:10:20.284 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-05-08T17:10:20.285 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-05-08T17:10:20.285 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-05-08T17:10:20.285 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-08T17:10:20.285 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-08T17:10:20.285 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-08T17:10:20.285 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T17:10:20.285 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81110 2024-05-08T17:10:20.285 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.81110/$cluster-$name.asok' 2024-05-08T17:10:20.287 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.81110/$cluster-$name.asok' 2024-05-08T17:10:20.287 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-05-08T17:10:20.287 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-05-08T17:10:20.287 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-05-08T17:10:20.287 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-05-08T17:10:20.287 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-05-08T17:10:20.287 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-08T17:10:20.287 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-08T17:10:20.287 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-08T17:10:20.287 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-08T17:10:20.287 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-05-08T17:10:20.287 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-05-08T17:10:20.288 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/2 2024-05-08T17:10:20.288 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-05-08T17:10:20.288 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=bbb34529-f998-4ab0-9482-f7a3ac16d4ec 2024-05-08T17:10:20.288 INFO:tasks.workunit.client.0.smithi172.stdout:add osd2 bbb34529-f998-4ab0-9482-f7a3ac16d4ec 2024-05-08T17:10:20.288 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 bbb34529-f998-4ab0-9482-f7a3ac16d4ec' 2024-05-08T17:10:20.288 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-05-08T17:10:20.310 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQD8sTtms+HiEhAAIRBm9aqChnHSHiso4Ff17Q== 2024-05-08T17:10:20.310 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQD8sTtms+HiEhAAIRBm9aqChnHSHiso4Ff17Q=="}' 2024-05-08T17:10:20.310 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new bbb34529-f998-4ab0-9482-f7a3ac16d4ec -i td/test-erasure-eio/2/new.json 2024-05-08T17:10:20.781 INFO:tasks.workunit.client.0.smithi172.stdout:2 2024-05-08T17:10:20.797 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/2/new.json 2024-05-08T17:10:20.798 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=4457b0f1-028b-482b-8ee3-7ee05861e6cc --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81110/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQD8sTtms+HiEhAAIRBm9aqChnHSHiso4Ff17Q== --osd-uuid bbb34529-f998-4ab0-9482-f7a3ac16d4ec 2024-05-08T17:10:20.831 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:10:20.838+0000 7fc17bcee7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T17:10:20.833 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:10:20.838+0000 7fc17bcee7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T17:10:20.835 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:10:20.838+0000 7fc17bcee7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T17:10:20.835 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:10:20.838+0000 7fc17bcee7c0 -1 bluestore(td/test-erasure-eio/2/block) _read_bdev_label failed to open td/test-erasure-eio/2/block: (2) No such file or directory 2024-05-08T17:10:20.835 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:10:20.838+0000 7fc17bcee7c0 -1 bluestore(td/test-erasure-eio/2/block) _read_bdev_label failed to open td/test-erasure-eio/2/block: (2) No such file or directory 2024-05-08T17:10:20.835 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:10:20.838+0000 7fc17bcee7c0 -1 bluestore(td/test-erasure-eio/2/block) _read_bdev_label failed to open td/test-erasure-eio/2/block: (2) No such file or directory 2024-05-08T17:10:20.928 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:10:20.934+0000 7fc17bcee7c0 -1 bluestore(td/test-erasure-eio/2) _read_fsid unparsable uuid 2024-05-08T17:10:28.101 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/2/keyring 2024-05-08T17:10:28.101 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-05-08T17:10:28.102 INFO:tasks.workunit.client.0.smithi172.stdout:adding osd2 key to auth repository 2024-05-08T17:10:28.102 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-05-08T17:10:28.102 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-05-08T17:10:28.655 INFO:tasks.workunit.client.0.smithi172.stdout:start osd.2 2024-05-08T17:10:28.656 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-05-08T17:10:28.656 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=4457b0f1-028b-482b-8ee3-7ee05861e6cc --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81110/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-05-08T17:10:28.656 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-05-08T17:10:28.657 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-05-08T17:10:28.657 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-05-08T17:10:28.681 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:10:28.686+0000 7f82c377b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T17:10:28.683 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:10:28.690+0000 7f82c377b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T17:10:28.685 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:10:28.690+0000 7f82c377b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T17:10:29.068 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-05-08T17:10:29.068 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-08T17:10:29.068 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-05-08T17:10:29.068 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-08T17:10:29.068 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-08T17:10:29.068 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:10:29.069 INFO:tasks.workunit.client.0.smithi172.stdout:0 2024-05-08T17:10:29.069 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-08T17:10:29.069 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:10:29.069 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T17:10:29.225 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:10:29.230+0000 7f82c377b7c0 -1 Falling back to public interface 2024-05-08T17:10:29.500 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:10:30.502 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:10:30.502 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:10:30.502 INFO:tasks.workunit.client.0.smithi172.stdout:1 2024-05-08T17:10:30.503 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-08T17:10:30.503 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:10:30.503 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T17:10:30.907 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:10:31.755 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:10:31.757+0000 7f82c377b7c0 -1 osd.2 0 log_to_monitors true 2024-05-08T17:10:31.908 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:10:31.909 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:10:31.909 INFO:tasks.workunit.client.0.smithi172.stdout:2 2024-05-08T17:10:31.909 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-08T17:10:31.909 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:10:31.909 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T17:10:32.579 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:10:33.581 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:10:33.581 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:10:33.581 INFO:tasks.workunit.client.0.smithi172.stdout:3 2024-05-08T17:10:33.582 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-05-08T17:10:33.582 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:10:33.582 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T17:10:34.128 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:10:35.130 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:10:35.130 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:10:35.130 INFO:tasks.workunit.client.0.smithi172.stdout:4 2024-05-08T17:10:35.131 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-05-08T17:10:35.131 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:10:35.131 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T17:10:35.870 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:10:36.872 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:10:36.872 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:10:36.872 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-05-08T17:10:36.873 INFO:tasks.workunit.client.0.smithi172.stdout:5 2024-05-08T17:10:36.873 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:10:36.873 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T17:10:37.446 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:10:38.448 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:10:38.448 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:10:38.448 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-05-08T17:10:38.448 INFO:tasks.workunit.client.0.smithi172.stdout:6 2024-05-08T17:10:38.449 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:10:38.449 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T17:10:39.040 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:10:40.042 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:10:40.042 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:10:40.042 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-05-08T17:10:40.042 INFO:tasks.workunit.client.0.smithi172.stdout:7 2024-05-08T17:10:40.043 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:10:40.043 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T17:10:40.465 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:10:41.467 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:10:41.467 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:10:41.467 INFO:tasks.workunit.client.0.smithi172.stdout:8 2024-05-08T17:10:41.468 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-05-08T17:10:41.468 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:10:41.468 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T17:10:41.947 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:10:42.948 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:10:42.949 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:10:42.949 INFO:tasks.workunit.client.0.smithi172.stdout:9 2024-05-08T17:10:42.949 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-05-08T17:10:42.950 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:10:42.950 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T17:10:43.474 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:10:44.475 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:10:44.476 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:10:44.476 INFO:tasks.workunit.client.0.smithi172.stdout:10 2024-05-08T17:10:44.476 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-05-08T17:10:44.476 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:10:44.477 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T17:10:45.080 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:10:46.082 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:10:46.082 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:10:46.083 INFO:tasks.workunit.client.0.smithi172.stdout:11 2024-05-08T17:10:46.083 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-05-08T17:10:46.083 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:10:46.083 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T17:10:46.741 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:10:47.743 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:10:47.743 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:10:47.743 INFO:tasks.workunit.client.0.smithi172.stdout:12 2024-05-08T17:10:47.744 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-05-08T17:10:47.744 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:10:47.744 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T17:10:48.511 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:10:49.513 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:10:49.513 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:10:49.513 INFO:tasks.workunit.client.0.smithi172.stdout:13 2024-05-08T17:10:49.514 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-05-08T17:10:49.514 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:10:49.514 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T17:10:50.465 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:10:51.467 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:10:51.467 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:10:51.467 INFO:tasks.workunit.client.0.smithi172.stdout:14 2024-05-08T17:10:51.468 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-05-08T17:10:51.468 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:10:51.468 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T17:10:52.594 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:10:53.596 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:10:53.596 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:10:53.597 INFO:tasks.workunit.client.0.smithi172.stdout:15 2024-05-08T17:10:53.597 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-05-08T17:10:53.597 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:10:53.597 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T17:10:54.562 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:10:55.564 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:10:55.564 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:10:55.565 INFO:tasks.workunit.client.0.smithi172.stdout:16 2024-05-08T17:10:55.565 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-05-08T17:10:55.565 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:10:55.565 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T17:10:56.384 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:10:57.386 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:10:57.387 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:10:57.388 INFO:tasks.workunit.client.0.smithi172.stdout:17 2024-05-08T17:10:57.388 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-05-08T17:10:57.388 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:10:57.388 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T17:10:58.352 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:10:59.354 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:10:59.354 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:10:59.354 INFO:tasks.workunit.client.0.smithi172.stdout:18 2024-05-08T17:10:59.355 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-05-08T17:10:59.355 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:10:59.355 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T17:10:59.777 INFO:tasks.workunit.client.0.smithi172.stdout:osd.2 up in weight 1 up_from 18 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/2776705647,v1:127.0.0.1:6819/2776705647] [v2:127.0.0.1:6820/2776705647,v1:127.0.0.1:6821/2776705647] exists,up bbb34529-f998-4ab0-9482-f7a3ac16d4ec 2024-05-08T17:10:59.779 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-08T17:10:59.779 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-08T17:10:59.779 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-08T17:10:59.779 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-05-08T17:10:59.779 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 3 2024-05-08T17:10:59.779 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-05-08T17:10:59.779 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-05-08T17:10:59.779 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2024-05-08T17:10:59.779 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-05-08T17:10:59.779 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/3 2024-05-08T17:10:59.779 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=4457b0f1-028b-482b-8ee3-7ee05861e6cc --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-05-08T17:10:59.779 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-08T17:10:59.780 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-05-08T17:10:59.780 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-08T17:10:59.780 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/3' 2024-05-08T17:10:59.780 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/3/journal' 2024-05-08T17:10:59.780 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-05-08T17:10:59.780 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-05-08T17:10:59.780 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-05-08T17:10:59.780 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-05-08T17:10:59.780 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-08T17:10:59.780 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-08T17:10:59.780 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-08T17:10:59.780 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T17:10:59.780 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81110 2024-05-08T17:10:59.780 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.81110/$cluster-$name.asok' 2024-05-08T17:10:59.782 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.81110/$cluster-$name.asok' 2024-05-08T17:10:59.782 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-05-08T17:10:59.782 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-05-08T17:10:59.782 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-05-08T17:10:59.782 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-05-08T17:10:59.782 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-05-08T17:10:59.782 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-08T17:10:59.782 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-08T17:10:59.782 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-08T17:10:59.782 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-08T17:10:59.782 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-05-08T17:10:59.783 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-05-08T17:10:59.783 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/3 2024-05-08T17:10:59.783 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-05-08T17:10:59.783 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=2280b00b-a008-445f-8af8-65406b10a936 2024-05-08T17:10:59.783 INFO:tasks.workunit.client.0.smithi172.stdout:add osd3 2280b00b-a008-445f-8af8-65406b10a936 2024-05-08T17:10:59.783 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 2280b00b-a008-445f-8af8-65406b10a936' 2024-05-08T17:10:59.783 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-05-08T17:10:59.804 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAjsjtmDyNaMBAAUeGFNHBiBb9CMMuGQ3yS2w== 2024-05-08T17:10:59.804 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAjsjtmDyNaMBAAUeGFNHBiBb9CMMuGQ3yS2w=="}' 2024-05-08T17:10:59.805 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 2280b00b-a008-445f-8af8-65406b10a936 -i td/test-erasure-eio/3/new.json 2024-05-08T17:11:00.919 INFO:tasks.workunit.client.0.smithi172.stdout:3 2024-05-08T17:11:00.941 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/3/new.json 2024-05-08T17:11:00.943 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=4457b0f1-028b-482b-8ee3-7ee05861e6cc --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81110/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAjsjtmDyNaMBAAUeGFNHBiBb9CMMuGQ3yS2w== --osd-uuid 2280b00b-a008-445f-8af8-65406b10a936 2024-05-08T17:11:00.977 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:11:00.981+0000 7f7a793de7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T17:11:00.999 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:11:01.001+0000 7f7a793de7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T17:11:01.000 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:11:01.005+0000 7f7a793de7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T17:11:01.001 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:11:01.005+0000 7f7a793de7c0 -1 bluestore(td/test-erasure-eio/3/block) _read_bdev_label failed to open td/test-erasure-eio/3/block: (2) No such file or directory 2024-05-08T17:11:01.001 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:11:01.005+0000 7f7a793de7c0 -1 bluestore(td/test-erasure-eio/3/block) _read_bdev_label failed to open td/test-erasure-eio/3/block: (2) No such file or directory 2024-05-08T17:11:01.001 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:11:01.005+0000 7f7a793de7c0 -1 bluestore(td/test-erasure-eio/3/block) _read_bdev_label failed to open td/test-erasure-eio/3/block: (2) No such file or directory 2024-05-08T17:11:01.151 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:11:01.157+0000 7f7a793de7c0 -1 bluestore(td/test-erasure-eio/3) _read_fsid unparsable uuid 2024-05-08T17:11:08.433 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/3/keyring 2024-05-08T17:11:08.433 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-05-08T17:11:08.434 INFO:tasks.workunit.client.0.smithi172.stdout:adding osd3 key to auth repository 2024-05-08T17:11:08.435 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2024-05-08T17:11:08.435 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-05-08T17:11:08.974 INFO:tasks.workunit.client.0.smithi172.stdout:start osd.3 2024-05-08T17:11:08.974 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2024-05-08T17:11:08.974 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=4457b0f1-028b-482b-8ee3-7ee05861e6cc --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81110/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-05-08T17:11:08.975 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-05-08T17:11:08.975 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-05-08T17:11:08.975 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-05-08T17:11:08.999 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:11:09.000+0000 7fa3c8ed47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T17:11:09.001 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:11:09.004+0000 7fa3c8ed47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T17:11:09.003 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:11:09.004+0000 7fa3c8ed47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T17:11:09.420 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2024-05-08T17:11:09.420 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-08T17:11:09.420 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-05-08T17:11:09.420 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-08T17:11:09.420 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-08T17:11:09.420 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:11:09.420 INFO:tasks.workunit.client.0.smithi172.stdout:0 2024-05-08T17:11:09.421 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-08T17:11:09.421 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:11:09.421 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T17:11:09.565 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:11:09.568+0000 7fa3c8ed47c0 -1 Falling back to public interface 2024-05-08T17:11:09.825 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:11:10.827 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:11:10.827 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:11:10.827 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-08T17:11:10.827 INFO:tasks.workunit.client.0.smithi172.stdout:1 2024-05-08T17:11:10.828 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:11:10.828 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T17:11:11.244 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:11:12.008 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:11:12.012+0000 7fa3c8ed47c0 -1 osd.3 0 log_to_monitors true 2024-05-08T17:11:12.245 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:11:12.245 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:11:12.245 INFO:tasks.workunit.client.0.smithi172.stdout:2 2024-05-08T17:11:12.246 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-08T17:11:12.246 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:11:12.246 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T17:11:12.854 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:11:13.856 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:11:13.856 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:11:13.856 INFO:tasks.workunit.client.0.smithi172.stdout:3 2024-05-08T17:11:13.857 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-05-08T17:11:13.857 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:11:13.857 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T17:11:14.500 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:11:15.502 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:11:15.502 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:11:15.502 INFO:tasks.workunit.client.0.smithi172.stdout:4 2024-05-08T17:11:15.503 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-05-08T17:11:15.503 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:11:15.503 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T17:11:16.137 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:11:17.139 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:11:17.139 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:11:17.140 INFO:tasks.workunit.client.0.smithi172.stdout:5 2024-05-08T17:11:17.140 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-05-08T17:11:17.140 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:11:17.140 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T17:11:17.564 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:11:18.567 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:11:18.567 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:11:18.567 INFO:tasks.workunit.client.0.smithi172.stdout:6 2024-05-08T17:11:18.568 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-05-08T17:11:18.568 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:11:18.568 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T17:11:19.228 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:11:20.229 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:11:20.230 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:11:20.230 INFO:tasks.workunit.client.0.smithi172.stdout:7 2024-05-08T17:11:20.230 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-05-08T17:11:20.231 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:11:20.231 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T17:11:20.637 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:11:21.639 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:11:21.639 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:11:21.639 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-05-08T17:11:21.640 INFO:tasks.workunit.client.0.smithi172.stdout:8 2024-05-08T17:11:21.640 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:11:21.640 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T17:11:22.204 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:11:23.206 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:11:23.206 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:11:23.206 INFO:tasks.workunit.client.0.smithi172.stdout:9 2024-05-08T17:11:23.207 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-05-08T17:11:23.207 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:11:23.207 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T17:11:23.761 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:11:24.763 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:11:24.763 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:11:24.763 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-05-08T17:11:24.763 INFO:tasks.workunit.client.0.smithi172.stdout:10 2024-05-08T17:11:24.764 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:11:24.764 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T17:11:25.451 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:11:26.453 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:11:26.453 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:11:26.453 INFO:tasks.workunit.client.0.smithi172.stdout:11 2024-05-08T17:11:26.454 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-05-08T17:11:26.454 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:11:26.454 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T17:11:26.999 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:11:28.001 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:11:28.001 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:11:28.001 INFO:tasks.workunit.client.0.smithi172.stdout:12 2024-05-08T17:11:28.002 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-05-08T17:11:28.002 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:11:28.002 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T17:11:28.530 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:11:29.532 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:11:29.532 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:11:29.533 INFO:tasks.workunit.client.0.smithi172.stdout:13 2024-05-08T17:11:29.533 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-05-08T17:11:29.533 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:11:29.533 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T17:11:30.684 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:11:31.686 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:11:31.687 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:11:31.687 INFO:tasks.workunit.client.0.smithi172.stdout:14 2024-05-08T17:11:31.687 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-05-08T17:11:31.687 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:11:31.688 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T17:11:32.582 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:11:33.583 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:11:33.583 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:11:33.584 INFO:tasks.workunit.client.0.smithi172.stdout:15 2024-05-08T17:11:33.584 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-05-08T17:11:33.584 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:11:33.584 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T17:11:34.449 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:11:35.451 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:11:35.451 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:11:35.451 INFO:tasks.workunit.client.0.smithi172.stdout:16 2024-05-08T17:11:35.452 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-05-08T17:11:35.452 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:11:35.452 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T17:11:36.166 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:11:37.168 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:11:37.168 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:11:37.169 INFO:tasks.workunit.client.0.smithi172.stdout:17 2024-05-08T17:11:37.169 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-05-08T17:11:37.169 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:11:37.169 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T17:11:38.245 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:11:39.247 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:11:39.247 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:11:39.247 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-05-08T17:11:39.248 INFO:tasks.workunit.client.0.smithi172.stdout:18 2024-05-08T17:11:39.248 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:11:39.248 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T17:11:39.754 INFO:tasks.workunit.client.0.smithi172.stdout:osd.3 up in weight 1 up_from 27 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/2097509417,v1:127.0.0.1:6827/2097509417] [v2:127.0.0.1:6828/2097509417,v1:127.0.0.1:6829/2097509417] exists,up 2280b00b-a008-445f-8af8-65406b10a936 2024-05-08T17:11:39.755 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-08T17:11:39.755 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-08T17:11:39.755 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-08T17:11:39.756 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: get_asok_path osd.0 2024-05-08T17:11:39.756 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-05-08T17:11:39.756 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-05-08T17:11:39.756 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-08T17:11:39.756 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T17:11:39.757 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81110 2024-05-08T17:11:39.757 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.81110/ceph-osd.0.asok 2024-05-08T17:11:39.757 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: CEPH_ARGS= 2024-05-08T17:11:39.757 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: ceph --admin-daemon /tmp/ceph-asok.81110/ceph-osd.0.asok log flush 2024-05-08T17:11:39.863 INFO:tasks.workunit.client.0.smithi172.stdout:{} 2024-05-08T17:11:39.876 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:56: setup_osds: grep 'load: jerasure.*lrc' td/test-erasure-eio/osd.0.log 2024-05-08T17:11:39.882 INFO:tasks.workunit.client.0.smithi172.stdout:2024-05-08T17:09:11.876+0000 7f7aef3647c0 0 load: jerasure load: lrc 2024-05-08T17:11:39.883 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:270: TEST_rados_get_subread_eio_shard_1: local poolname=pool-jerasure 2024-05-08T17:11:39.883 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:271: TEST_rados_get_subread_eio_shard_1: create_erasure_coded_pool pool-jerasure 2 1 2024-05-08T17:11:39.883 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:67: create_erasure_coded_pool: local poolname=pool-jerasure 2024-05-08T17:11:39.883 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:68: create_erasure_coded_pool: shift 2024-05-08T17:11:39.883 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:69: create_erasure_coded_pool: local k=2 2024-05-08T17:11:39.883 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:70: create_erasure_coded_pool: shift 2024-05-08T17:11:39.883 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:71: create_erasure_coded_pool: local m=1 2024-05-08T17:11:39.883 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:72: create_erasure_coded_pool: shift 2024-05-08T17:11:39.883 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:74: create_erasure_coded_pool: ceph osd erasure-code-profile set myprofile plugin=jerasure k=2 m=1 crush-failure-domain=osd 2024-05-08T17:11:41.129 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:78: create_erasure_coded_pool: create_pool pool-jerasure 1 1 erasure myprofile 2024-05-08T17:11:41.129 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create pool-jerasure 1 1 erasure myprofile 2024-05-08T17:11:42.481 INFO:tasks.workunit.client.0.smithi172.stderr:pool 'pool-jerasure' already exists 2024-05-08T17:11:42.499 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-05-08T17:11:43.502 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:80: create_erasure_coded_pool: wait_for_clean 2024-05-08T17:11:43.502 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-05-08T17:11:43.502 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-05-08T17:11:43.502 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-05-08T17:11:43.502 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-05-08T17:11:43.504 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-05-08T17:11:43.504 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-05-08T17:11:43.504 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-05-08T17:11:43.504 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-05-08T17:11:43.504 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-05-08T17:11:43.636 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-05-08T17:11:43.636 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-05-08T17:11:43.636 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-05-08T17:11:43.636 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-05-08T17:11:43.636 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-05-08T17:11:43.636 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-05-08T17:11:44.163 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-05-08T17:11:44.163 INFO:tasks.workunit.client.0.smithi172.stderr:1 2024-05-08T17:11:44.163 INFO:tasks.workunit.client.0.smithi172.stderr:2 2024-05-08T17:11:44.163 INFO:tasks.workunit.client.0.smithi172.stderr:3' 2024-05-08T17:11:44.163 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-05-08T17:11:44.163 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T17:11:44.164 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-05-08T17:11:44.358 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=25769803808 2024-05-08T17:11:44.358 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 25769803808 2024-05-08T17:11:44.358 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803808' 2024-05-08T17:11:44.358 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T17:11:44.358 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-05-08T17:11:44.510 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=55834574873 2024-05-08T17:11:44.510 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 55834574873 2024-05-08T17:11:44.510 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803808 1-55834574873' 2024-05-08T17:11:44.511 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T17:11:44.511 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-05-08T17:11:44.653 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=77309411346 2024-05-08T17:11:44.653 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 77309411346 2024-05-08T17:11:44.653 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803808 1-55834574873 2-77309411346' 2024-05-08T17:11:44.653 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T17:11:44.653 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-05-08T17:11:44.794 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=115964117000 2024-05-08T17:11:44.794 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 115964117000 2024-05-08T17:11:44.795 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803808 1-55834574873 2-77309411346 3-115964117000' 2024-05-08T17:11:44.795 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-08T17:11:44.795 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-25769803808 2024-05-08T17:11:44.795 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-08T17:11:44.796 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-05-08T17:11:44.797 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-25769803808 2024-05-08T17:11:44.797 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-08T17:11:44.798 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=25769803808 2024-05-08T17:11:44.798 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 25769803808' 2024-05-08T17:11:44.799 INFO:tasks.workunit.client.0.smithi172.stdout:waiting osd.0 seq 25769803808 2024-05-08T17:11:44.799 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-05-08T17:11:45.189 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 25769803808 -lt 25769803808 2024-05-08T17:11:45.189 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-08T17:11:45.189 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-55834574873 2024-05-08T17:11:45.190 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-08T17:11:45.191 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-05-08T17:11:45.192 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-55834574873 2024-05-08T17:11:45.192 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-08T17:11:45.193 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=55834574873 2024-05-08T17:11:45.193 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 55834574873' 2024-05-08T17:11:45.194 INFO:tasks.workunit.client.0.smithi172.stdout:waiting osd.1 seq 55834574873 2024-05-08T17:11:45.194 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-05-08T17:11:45.615 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 55834574873 -lt 55834574873 2024-05-08T17:11:45.616 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-08T17:11:45.616 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-77309411346 2024-05-08T17:11:45.617 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-08T17:11:45.619 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-05-08T17:11:45.620 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-77309411346 2024-05-08T17:11:45.620 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-08T17:11:45.621 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=77309411346 2024-05-08T17:11:45.621 INFO:tasks.workunit.client.0.smithi172.stdout:waiting osd.2 seq 77309411346 2024-05-08T17:11:45.621 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 77309411346' 2024-05-08T17:11:45.622 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-05-08T17:11:46.035 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 77309411345 -lt 77309411346 2024-05-08T17:11:46.036 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-05-08T17:11:47.037 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-05-08T17:11:47.038 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-05-08T17:11:47.438 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 77309411346 -lt 77309411346 2024-05-08T17:11:47.439 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-08T17:11:47.439 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-115964117000 2024-05-08T17:11:47.439 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-08T17:11:47.441 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-05-08T17:11:47.442 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-115964117000 2024-05-08T17:11:47.442 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-08T17:11:47.443 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=115964117000 2024-05-08T17:11:47.444 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 115964117000' 2024-05-08T17:11:47.444 INFO:tasks.workunit.client.0.smithi172.stdout:waiting osd.3 seq 115964117000 2024-05-08T17:11:47.444 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-05-08T17:11:47.866 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 115964117000 -lt 115964117000 2024-05-08T17:11:47.867 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-05-08T17:11:47.867 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-05-08T17:11:47.868 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-05-08T17:11:48.386 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-05-08T17:11:48.386 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-05-08T17:11:48.387 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-05-08T17:11:48.387 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-05-08T17:11:48.387 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-05-08T17:11:48.387 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-05-08T17:11:48.387 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-05-08T17:11:48.387 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-05-08T17:11:48.856 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-05-08T17:11:48.857 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-05-08T17:11:48.857 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-05-08T17:11:48.857 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-05-08T17:11:49.426 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-05-08T17:11:49.426 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-05-08T17:11:49.426 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-05-08T17:11:49.426 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:273: TEST_rados_get_subread_eio_shard_1: local shard_id=1 2024-05-08T17:11:49.426 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:274: TEST_rados_get_subread_eio_shard_1: rados_put_get_data eio td/test-erasure-eio 1 2024-05-08T17:11:49.426 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:147: rados_put_get_data: local inject=eio 2024-05-08T17:11:49.426 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:148: rados_put_get_data: shift 2024-05-08T17:11:49.426 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:149: rados_put_get_data: local dir=td/test-erasure-eio 2024-05-08T17:11:49.427 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:150: rados_put_get_data: shift 2024-05-08T17:11:49.427 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:151: rados_put_get_data: local shard_id=1 2024-05-08T17:11:49.427 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:152: rados_put_get_data: shift 2024-05-08T17:11:49.427 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:153: rados_put_get_data: local arg= 2024-05-08T17:11:49.427 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:157: rados_put_get_data: local poolname=pool-jerasure 2024-05-08T17:11:49.427 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:158: rados_put_get_data: local objname=obj-eio-81110-1 2024-05-08T17:11:49.427 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:159: rados_put_get_data: rados_put td/test-erasure-eio pool-jerasure obj-eio-81110-1 2024-05-08T17:11:49.427 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:90: rados_put: local dir=td/test-erasure-eio 2024-05-08T17:11:49.427 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:91: rados_put: local poolname=pool-jerasure 2024-05-08T17:11:49.427 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:92: rados_put: local objname=obj-eio-81110-1 2024-05-08T17:11:49.427 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-05-08T17:11:49.427 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 AAA 2024-05-08T17:11:49.427 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-05-08T17:11:49.427 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 BBB 2024-05-08T17:11:49.427 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-05-08T17:11:49.428 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 CCCC 2024-05-08T17:11:49.428 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-05-08T17:11:49.428 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 DDDD 2024-05-08T17:11:49.428 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:100: rados_put: rados --pool pool-jerasure put obj-eio-81110-1 td/test-erasure-eio/ORIGINAL 2024-05-08T17:11:49.618 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:160: rados_put_get_data: inject_eio ec data pool-jerasure obj-eio-81110-1 td/test-erasure-eio 1 2024-05-08T17:11:49.618 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2394: inject_eio: local pooltype=ec 2024-05-08T17:11:49.619 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2395: inject_eio: shift 2024-05-08T17:11:49.619 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2396: inject_eio: local which=data 2024-05-08T17:11:49.619 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2397: inject_eio: shift 2024-05-08T17:11:49.619 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2398: inject_eio: local poolname=pool-jerasure 2024-05-08T17:11:49.619 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2399: inject_eio: shift 2024-05-08T17:11:49.619 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2400: inject_eio: local objname=obj-eio-81110-1 2024-05-08T17:11:49.619 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2401: inject_eio: shift 2024-05-08T17:11:49.619 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2402: inject_eio: local dir=td/test-erasure-eio 2024-05-08T17:11:49.619 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2403: inject_eio: shift 2024-05-08T17:11:49.619 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2404: inject_eio: local shard_id=1 2024-05-08T17:11:49.619 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2405: inject_eio: shift 2024-05-08T17:11:49.619 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: get_osds pool-jerasure obj-eio-81110-1 2024-05-08T17:11:49.619 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2024-05-08T17:11:49.620 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-eio-81110-1 2024-05-08T17:11:49.620 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure obj-eio-81110-1 2024-05-08T17:11:49.620 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-05-08T17:11:50.087 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-05-08T17:11:50.087 INFO:tasks.workunit.client.0.smithi172.stderr:1 2024-05-08T17:11:50.087 INFO:tasks.workunit.client.0.smithi172.stderr:2' 2024-05-08T17:11:50.087 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 2 2024-05-08T17:11:50.087 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: initial_osds=('3' '1' '2') 2024-05-08T17:11:50.087 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: local -a initial_osds 2024-05-08T17:11:50.087 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2408: inject_eio: local osd_id=1 2024-05-08T17:11:50.087 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2409: inject_eio: '[' ec '!=' ec ']' 2024-05-08T17:11:50.088 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: cat td/test-erasure-eio/1/type 2024-05-08T17:11:50.089 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: type=bluestore 2024-05-08T17:11:50.089 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2024-05-08T17:11:50.089 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-05-08T17:11:50.089 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=1 2024-05-08T17:11:50.089 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-05-08T17:11:50.089 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-05-08T17:11:50.091 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-05-08T17:11:50.091 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.1 2024-05-08T17:11:50.091 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-05-08T17:11:50.091 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-05-08T17:11:50.091 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-08T17:11:50.091 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T17:11:50.092 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81110 2024-05-08T17:11:50.092 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.81110/ceph-osd.1.asok 2024-05-08T17:11:50.092 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.81110/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2024-05-08T17:11:50.214 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-05-08T17:11:50.214 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local loop=0 2024-05-08T17:11:50.215 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: grep -q Invalid 2024-05-08T17:11:50.215 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: get_asok_path osd.1 2024-05-08T17:11:50.215 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-05-08T17:11:50.215 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-05-08T17:11:50.216 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-08T17:11:50.216 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T17:11:50.216 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81110 2024-05-08T17:11:50.216 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.81110/ceph-osd.1.asok 2024-05-08T17:11:50.216 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: CEPH_ARGS= 2024-05-08T17:11:50.217 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: ceph --admin-daemon /tmp/ceph-asok.81110/ceph-osd.1.asok injectdataerr pool-jerasure obj-eio-81110-1 1 2024-05-08T17:11:50.341 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:161: rados_put_get_data: rados_get td/test-erasure-eio pool-jerasure obj-eio-81110-1 2024-05-08T17:11:50.342 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2024-05-08T17:11:50.342 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2024-05-08T17:11:50.342 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=obj-eio-81110-1 2024-05-08T17:11:50.342 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=ok 2024-05-08T17:11:50.342 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' ok = fail ']' 2024-05-08T17:11:50.342 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:120: rados_get: rados --pool pool-jerasure get obj-eio-81110-1 td/test-erasure-eio/COPY 2024-05-08T17:11:50.380 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:121: rados_get: diff td/test-erasure-eio/ORIGINAL td/test-erasure-eio/COPY 2024-05-08T17:11:50.381 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:122: rados_get: rm td/test-erasure-eio/COPY 2024-05-08T17:11:50.381 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:163: rados_put_get_data: '[' '' = recovery ']' 2024-05-08T17:11:50.382 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:181: rados_put_get_data: expr 1 + 1 2024-05-08T17:11:50.384 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:181: rados_put_get_data: shard_id=2 2024-05-08T17:11:50.384 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:182: rados_put_get_data: inject_eio ec data pool-jerasure obj-eio-81110-1 td/test-erasure-eio 2 2024-05-08T17:11:50.385 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2394: inject_eio: local pooltype=ec 2024-05-08T17:11:50.385 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2395: inject_eio: shift 2024-05-08T17:11:50.385 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2396: inject_eio: local which=data 2024-05-08T17:11:50.385 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2397: inject_eio: shift 2024-05-08T17:11:50.385 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2398: inject_eio: local poolname=pool-jerasure 2024-05-08T17:11:50.385 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2399: inject_eio: shift 2024-05-08T17:11:50.385 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2400: inject_eio: local objname=obj-eio-81110-1 2024-05-08T17:11:50.385 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2401: inject_eio: shift 2024-05-08T17:11:50.385 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2402: inject_eio: local dir=td/test-erasure-eio 2024-05-08T17:11:50.385 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2403: inject_eio: shift 2024-05-08T17:11:50.385 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2404: inject_eio: local shard_id=2 2024-05-08T17:11:50.385 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2405: inject_eio: shift 2024-05-08T17:11:50.385 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: get_osds pool-jerasure obj-eio-81110-1 2024-05-08T17:11:50.385 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2024-05-08T17:11:50.385 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-eio-81110-1 2024-05-08T17:11:50.386 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure obj-eio-81110-1 2024-05-08T17:11:50.386 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-05-08T17:11:50.829 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-05-08T17:11:50.829 INFO:tasks.workunit.client.0.smithi172.stderr:1 2024-05-08T17:11:50.829 INFO:tasks.workunit.client.0.smithi172.stderr:2' 2024-05-08T17:11:50.829 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 2 2024-05-08T17:11:50.829 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: initial_osds=('3' '1' '2') 2024-05-08T17:11:50.830 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: local -a initial_osds 2024-05-08T17:11:50.830 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2408: inject_eio: local osd_id=2 2024-05-08T17:11:50.830 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2409: inject_eio: '[' ec '!=' ec ']' 2024-05-08T17:11:50.831 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: cat td/test-erasure-eio/2/type 2024-05-08T17:11:50.831 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: type=bluestore 2024-05-08T17:11:50.831 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: set_config osd 2 bluestore_debug_inject_read_err true 2024-05-08T17:11:50.831 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-05-08T17:11:50.831 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=2 2024-05-08T17:11:50.831 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-05-08T17:11:50.831 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-05-08T17:11:50.832 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-05-08T17:11:50.832 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.2 2024-05-08T17:11:50.832 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2024-05-08T17:11:50.832 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2024-05-08T17:11:50.833 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-08T17:11:50.833 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T17:11:50.833 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81110 2024-05-08T17:11:50.833 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.81110/ceph-osd.2.asok 2024-05-08T17:11:50.834 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.81110/ceph-osd.2.asok config set bluestore_debug_inject_read_err true 2024-05-08T17:11:50.956 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-05-08T17:11:50.956 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local loop=0 2024-05-08T17:11:50.958 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: grep -q Invalid 2024-05-08T17:11:50.958 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: get_asok_path osd.2 2024-05-08T17:11:50.958 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2024-05-08T17:11:50.958 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2024-05-08T17:11:50.959 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-08T17:11:50.959 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T17:11:50.959 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81110 2024-05-08T17:11:50.959 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.81110/ceph-osd.2.asok 2024-05-08T17:11:50.960 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: CEPH_ARGS= 2024-05-08T17:11:50.960 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: ceph --admin-daemon /tmp/ceph-asok.81110/ceph-osd.2.asok injectdataerr pool-jerasure obj-eio-81110-1 2 2024-05-08T17:11:51.079 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:183: rados_put_get_data: rados_get td/test-erasure-eio pool-jerasure obj-eio-81110-1 fail 2024-05-08T17:11:51.079 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2024-05-08T17:11:51.079 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2024-05-08T17:11:51.079 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=obj-eio-81110-1 2024-05-08T17:11:51.079 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=fail 2024-05-08T17:11:51.079 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' fail = fail ']' 2024-05-08T17:11:51.079 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:114: rados_get: rados --pool pool-jerasure get obj-eio-81110-1 td/test-erasure-eio/COPY 2024-05-08T17:11:51.114 INFO:tasks.workunit.client.0.smithi172.stderr:error getting pool-jerasure/obj-eio-81110-1: (5) Input/output error 2024-05-08T17:11:51.116 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:115: rados_get: return 2024-05-08T17:11:51.116 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:184: rados_put_get_data: rm td/test-erasure-eio/ORIGINAL 2024-05-08T17:11:51.117 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:275: TEST_rados_get_subread_eio_shard_1: delete_erasure_coded_pool pool-jerasure 2024-05-08T17:11:51.117 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:84: delete_erasure_coded_pool: local poolname=pool-jerasure 2024-05-08T17:11:51.117 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:85: delete_erasure_coded_pool: ceph osd pool delete pool-jerasure pool-jerasure --yes-i-really-really-mean-it 2024-05-08T17:11:51.636 INFO:tasks.workunit.client.0.smithi172.stderr:pool 'pool-jerasure' does not exist 2024-05-08T17:11:51.652 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:86: delete_erasure_coded_pool: ceph osd erasure-code-profile rm myprofile 2024-05-08T17:11:52.348 INFO:tasks.workunit.client.0.smithi172.stderr:erasure-code-profile myprofile does not exist 2024-05-08T17:11:52.365 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:42: run: teardown td/test-erasure-eio 2024-05-08T17:11:52.365 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-eio 2024-05-08T17:11:52.365 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-05-08T17:11:52.366 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-eio KILL 2024-05-08T17:11:52.366 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-08T17:11:52.366 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-08T17:11:52.367 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-08T17:11:52.367 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-08T17:11:52.367 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-08T17:11:52.678 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-08T17:11:52.679 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-05-08T17:11:52.679 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-05-08T17:11:52.680 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-05-08T17:11:52.682 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-05-08T17:11:52.682 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-05-08T17:11:52.683 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-05-08T17:11:52.684 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-08T17:11:52.685 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-05-08T17:11:52.685 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-05-08T17:11:52.686 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-08T17:11:52.687 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-05-08T17:11:52.688 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-05-08T17:11:52.688 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-eio 2024-05-08T17:11:52.713 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-05-08T17:11:52.713 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T17:11:52.713 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81110 2024-05-08T17:11:52.713 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.81110 2024-05-08T17:11:52.715 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-05-08T17:11:52.715 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-05-08T17:11:52.715 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:32: run: for func in $funcs 2024-05-08T17:11:52.715 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:33: run: setup td/test-erasure-eio 2024-05-08T17:11:52.716 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/test-erasure-eio 2024-05-08T17:11:52.716 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/test-erasure-eio 2024-05-08T17:11:52.716 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-eio 2024-05-08T17:11:52.716 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-05-08T17:11:52.716 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-eio KILL 2024-05-08T17:11:52.716 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-08T17:11:52.716 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-08T17:11:52.716 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-08T17:11:52.716 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-08T17:11:52.716 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-08T17:11:52.718 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-08T17:11:52.719 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-05-08T17:11:52.719 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-05-08T17:11:52.720 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-05-08T17:11:52.720 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-05-08T17:11:52.720 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-05-08T17:11:52.721 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-05-08T17:11:52.723 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-08T17:11:52.724 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-05-08T17:11:52.724 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-05-08T17:11:52.724 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-08T17:11:52.725 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-05-08T17:11:52.725 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-05-08T17:11:52.725 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-eio 2024-05-08T17:11:52.726 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-05-08T17:11:52.726 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T17:11:52.727 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81110 2024-05-08T17:11:52.727 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.81110 2024-05-08T17:11:52.728 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-05-08T17:11:52.728 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-05-08T17:11:52.728 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/test-erasure-eio 2024-05-08T17:11:52.729 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-05-08T17:11:52.729 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T17:11:52.729 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81110 2024-05-08T17:11:52.729 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.81110 2024-05-08T17:11:52.732 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-05-08T17:11:52.732 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-05-08T17:11:52.732 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-05-08T17:11:52.732 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/test-erasure-eio 1' TERM HUP INT 2024-05-08T17:11:52.732 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:34: run: run_mon td/test-erasure-eio a 2024-05-08T17:11:52.732 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/test-erasure-eio 2024-05-08T17:11:52.732 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-05-08T17:11:52.732 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-05-08T17:11:52.732 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-05-08T17:11:52.732 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/test-erasure-eio/a 2024-05-08T17:11:52.732 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-eio/a --run-dir=td/test-erasure-eio 2024-05-08T17:11:53.132 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-05-08T17:11:53.132 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-08T17:11:53.132 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-08T17:11:53.133 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-08T17:11:53.133 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T17:11:53.133 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81110 2024-05-08T17:11:53.133 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.81110/$cluster-$name.asok' 2024-05-08T17:11:53.134 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-eio/a '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.81110/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-eio/log --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-05-08T17:11:53.341 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-05-08T17:11:53.343 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-05-08T17:11:53.343 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-05-08T17:11:53.343 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-05-08T17:11:53.343 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-05-08T17:11:53.344 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-05-08T17:11:53.344 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-05-08T17:11:53.344 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-05-08T17:11:53.344 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-05-08T17:11:53.345 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-08T17:11:53.345 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T17:11:53.345 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81110 2024-05-08T17:11:53.345 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.81110/ceph-mon.a.asok 2024-05-08T17:11:53.345 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-05-08T17:11:53.346 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.81110/ceph-mon.a.asok config get fsid 2024-05-08T17:11:53.462 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-05-08T17:11:53.463 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-05-08T17:11:53.463 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-05-08T17:11:53.463 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-05-08T17:11:53.463 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-05-08T17:11:53.464 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-05-08T17:11:53.464 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-05-08T17:11:53.464 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-05-08T17:11:53.465 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-08T17:11:53.465 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T17:11:53.465 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81110 2024-05-08T17:11:53.465 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.81110/ceph-mon.a.asok 2024-05-08T17:11:53.465 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-05-08T17:11:53.465 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.81110/ceph-mon.a.asok config get mon_host 2024-05-08T17:11:53.574 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:35: run: run_mgr td/test-erasure-eio x 2024-05-08T17:11:53.575 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/test-erasure-eio 2024-05-08T17:11:53.575 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-05-08T17:11:53.575 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-05-08T17:11:53.575 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-05-08T17:11:53.575 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/test-erasure-eio/x 2024-05-08T17:11:53.575 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-05-08T17:11:54.487 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-05-08T17:11:54.488 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-08T17:11:54.488 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-08T17:11:54.488 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-08T17:11:54.488 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T17:11:54.488 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81110 2024-05-08T17:11:54.488 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.81110/$cluster-$name.asok' 2024-05-08T17:11:54.489 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-05-08T17:11:54.491 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-eio/x '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.81110/$cluster-$name.asok' --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-05-08T17:11:54.521 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:36: run: create_pool rbd 4 2024-05-08T17:11:54.521 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-05-08T17:11:54.813 INFO:tasks.workunit.client.0.smithi172.stderr:pool 'rbd' already exists 2024-05-08T17:11:54.829 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-05-08T17:11:55.832 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: get_asok_path mon.a 2024-05-08T17:11:55.832 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-05-08T17:11:55.832 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-05-08T17:11:55.833 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-08T17:11:55.833 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T17:11:55.833 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81110 2024-05-08T17:11:55.833 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.81110/ceph-mon.a.asok 2024-05-08T17:11:55.833 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: CEPH_ARGS= 2024-05-08T17:11:55.833 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: ceph --admin-daemon /tmp/ceph-asok.81110/ceph-mon.a.asok log flush 2024-05-08T17:11:55.933 INFO:tasks.workunit.client.0.smithi172.stdout:{} 2024-05-08T17:11:55.945 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:40: run: grep 'load: jerasure.*lrc' td/test-erasure-eio/mon.a.log 2024-05-08T17:11:55.948 INFO:tasks.workunit.client.0.smithi172.stdout:2024-05-08T17:11:53.175+0000 7f09a5df0c80 0 load: jerasure load: lrc 2024-05-08T17:11:55.949 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:41: run: TEST_rados_get_subread_missing td/test-erasure-eio 2024-05-08T17:11:55.949 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:282: TEST_rados_get_subread_missing: local dir=td/test-erasure-eio 2024-05-08T17:11:55.949 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:283: TEST_rados_get_subread_missing: setup_osds 4 2024-05-08T17:11:55.949 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:47: setup_osds: local count=4 2024-05-08T17:11:55.949 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:48: setup_osds: shift 2024-05-08T17:11:55.950 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: expr 4 - 1 2024-05-08T17:11:55.952 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: seq 0 3 2024-05-08T17:11:55.955 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-05-08T17:11:55.955 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 0 2024-05-08T17:11:55.955 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-05-08T17:11:55.955 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-05-08T17:11:55.955 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-05-08T17:11:55.955 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-05-08T17:11:55.955 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/0 2024-05-08T17:11:55.955 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=4457b0f1-028b-482b-8ee3-7ee05861e6cc --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-05-08T17:11:55.955 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-08T17:11:55.956 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-05-08T17:11:55.956 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-08T17:11:55.956 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/0' 2024-05-08T17:11:55.956 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/0/journal' 2024-05-08T17:11:55.956 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-05-08T17:11:55.956 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-05-08T17:11:55.956 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-05-08T17:11:55.956 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-05-08T17:11:55.956 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-08T17:11:55.956 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-08T17:11:55.956 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-08T17:11:55.956 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T17:11:55.956 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81110 2024-05-08T17:11:55.956 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.81110/$cluster-$name.asok' 2024-05-08T17:11:55.957 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.81110/$cluster-$name.asok' 2024-05-08T17:11:55.957 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-05-08T17:11:55.957 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-05-08T17:11:55.957 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-05-08T17:11:55.957 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-05-08T17:11:55.957 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-05-08T17:11:55.957 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-08T17:11:55.957 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-08T17:11:55.957 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-08T17:11:55.957 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-08T17:11:55.957 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-05-08T17:11:55.957 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-05-08T17:11:55.957 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/0 2024-05-08T17:11:55.958 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-05-08T17:11:55.958 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=680298c1-f1f0-4efb-ba2b-6125284012aa 2024-05-08T17:11:55.958 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 680298c1-f1f0-4efb-ba2b-6125284012aa' 2024-05-08T17:11:55.958 INFO:tasks.workunit.client.0.smithi172.stdout:add osd0 680298c1-f1f0-4efb-ba2b-6125284012aa 2024-05-08T17:11:55.959 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-05-08T17:11:55.978 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBbsjtmj7G3OhAAey73UkW4xJL1ONGdlwT8eA== 2024-05-08T17:11:55.979 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBbsjtmj7G3OhAAey73UkW4xJL1ONGdlwT8eA=="}' 2024-05-08T17:11:55.979 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 680298c1-f1f0-4efb-ba2b-6125284012aa -i td/test-erasure-eio/0/new.json 2024-05-08T17:11:56.272 INFO:tasks.workunit.client.0.smithi172.stdout:0 2024-05-08T17:11:56.288 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/0/new.json 2024-05-08T17:11:56.289 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=4457b0f1-028b-482b-8ee3-7ee05861e6cc --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81110/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBbsjtmj7G3OhAAey73UkW4xJL1ONGdlwT8eA== --osd-uuid 680298c1-f1f0-4efb-ba2b-6125284012aa 2024-05-08T17:11:56.322 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:11:56.323+0000 7f0b54e417c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T17:11:56.340 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:11:56.343+0000 7f0b54e417c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T17:11:56.342 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:11:56.343+0000 7f0b54e417c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T17:11:56.342 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:11:56.343+0000 7f0b54e417c0 -1 bluestore(td/test-erasure-eio/0/block) _read_bdev_label failed to open td/test-erasure-eio/0/block: (2) No such file or directory 2024-05-08T17:11:56.342 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:11:56.343+0000 7f0b54e417c0 -1 bluestore(td/test-erasure-eio/0/block) _read_bdev_label failed to open td/test-erasure-eio/0/block: (2) No such file or directory 2024-05-08T17:11:56.342 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:11:56.343+0000 7f0b54e417c0 -1 bluestore(td/test-erasure-eio/0/block) _read_bdev_label failed to open td/test-erasure-eio/0/block: (2) No such file or directory 2024-05-08T17:11:56.381 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:11:56.383+0000 7f0b54e417c0 -1 bluestore(td/test-erasure-eio/0) _read_fsid unparsable uuid 2024-05-08T17:12:02.007 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/0/keyring 2024-05-08T17:12:02.008 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-05-08T17:12:02.009 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-05-08T17:12:02.009 INFO:tasks.workunit.client.0.smithi172.stdout:adding osd0 key to auth repository 2024-05-08T17:12:02.009 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-05-08T17:12:02.551 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-05-08T17:12:02.552 INFO:tasks.workunit.client.0.smithi172.stdout:start osd.0 2024-05-08T17:12:02.552 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=4457b0f1-028b-482b-8ee3-7ee05861e6cc --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81110/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-05-08T17:12:02.552 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-05-08T17:12:02.552 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-05-08T17:12:02.552 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-05-08T17:12:02.578 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:12:02.583+0000 7fdfec5687c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T17:12:02.580 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:12:02.587+0000 7fdfec5687c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T17:12:02.581 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:12:02.587+0000 7fdfec5687c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T17:12:02.973 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-05-08T17:12:02.974 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-08T17:12:02.974 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-05-08T17:12:02.974 INFO:tasks.workunit.client.0.smithi172.stdout:0 2024-05-08T17:12:02.974 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-08T17:12:02.974 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-08T17:12:02.975 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:12:02.975 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-08T17:12:02.975 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:12:02.975 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T17:12:03.149 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:12:03.155+0000 7fdfec5687c0 -1 Falling back to public interface 2024-05-08T17:12:03.393 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:12:04.395 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:12:04.395 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:12:04.395 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-08T17:12:04.396 INFO:tasks.workunit.client.0.smithi172.stdout:1 2024-05-08T17:12:04.396 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:12:04.396 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T17:12:04.795 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:12:05.624 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:12:05.631+0000 7fdfec5687c0 -1 osd.0 0 log_to_monitors true 2024-05-08T17:12:05.797 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:12:05.797 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:12:05.797 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-08T17:12:05.798 INFO:tasks.workunit.client.0.smithi172.stdout:2 2024-05-08T17:12:05.798 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:12:05.798 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T17:12:06.343 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:12:07.344 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:12:07.344 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:12:07.345 INFO:tasks.workunit.client.0.smithi172.stdout:3 2024-05-08T17:12:07.345 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-05-08T17:12:07.345 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:12:07.345 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T17:12:07.800 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:12:08.802 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:12:08.802 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:12:08.802 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-05-08T17:12:08.803 INFO:tasks.workunit.client.0.smithi172.stdout:4 2024-05-08T17:12:08.803 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:12:08.803 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T17:12:09.360 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:12:10.362 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:12:10.362 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:12:10.363 INFO:tasks.workunit.client.0.smithi172.stdout:5 2024-05-08T17:12:10.363 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-05-08T17:12:10.363 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:12:10.363 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T17:12:10.781 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:12:11.782 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:12:11.782 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:12:11.783 INFO:tasks.workunit.client.0.smithi172.stdout:6 2024-05-08T17:12:11.783 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-05-08T17:12:11.783 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:12:11.783 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T17:12:12.300 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:12:13.302 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:12:13.302 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:12:13.302 INFO:tasks.workunit.client.0.smithi172.stdout:7 2024-05-08T17:12:13.303 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-05-08T17:12:13.303 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:12:13.303 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T17:12:13.746 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:12:14.747 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:12:14.747 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:12:14.748 INFO:tasks.workunit.client.0.smithi172.stdout:8 2024-05-08T17:12:14.750 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-05-08T17:12:14.750 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:12:14.750 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T17:12:15.171 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:12:16.173 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:12:16.173 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:12:16.174 INFO:tasks.workunit.client.0.smithi172.stdout:9 2024-05-08T17:12:16.174 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-05-08T17:12:16.174 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:12:16.174 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T17:12:16.627 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:12:17.628 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:12:17.629 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:12:17.629 INFO:tasks.workunit.client.0.smithi172.stdout:10 2024-05-08T17:12:17.629 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-05-08T17:12:17.629 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:12:17.629 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T17:12:18.204 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:12:19.206 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:12:19.207 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:12:19.207 INFO:tasks.workunit.client.0.smithi172.stdout:11 2024-05-08T17:12:19.207 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-05-08T17:12:19.207 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:12:19.208 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T17:12:20.024 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:12:21.026 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:12:21.026 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:12:21.026 INFO:tasks.workunit.client.0.smithi172.stdout:12 2024-05-08T17:12:21.027 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-05-08T17:12:21.027 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:12:21.027 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T17:12:22.024 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:12:23.026 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:12:23.026 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:12:23.026 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-05-08T17:12:23.026 INFO:tasks.workunit.client.0.smithi172.stdout:13 2024-05-08T17:12:23.027 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:12:23.027 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T17:12:23.548 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:12:24.550 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:12:24.550 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:12:24.550 INFO:tasks.workunit.client.0.smithi172.stdout:14 2024-05-08T17:12:24.551 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-05-08T17:12:24.551 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:12:24.551 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T17:12:25.200 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:12:26.202 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:12:26.202 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:12:26.202 INFO:tasks.workunit.client.0.smithi172.stdout:15 2024-05-08T17:12:26.203 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-05-08T17:12:26.203 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:12:26.203 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T17:12:27.014 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:12:28.016 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:12:28.016 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:12:28.016 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-05-08T17:12:28.016 INFO:tasks.workunit.client.0.smithi172.stdout:16 2024-05-08T17:12:28.017 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:12:28.017 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T17:12:28.586 INFO:tasks.workunit.client.0.smithi172.stdout:osd.0 up in weight 1 up_from 6 up_thru 6 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/165531318,v1:127.0.0.1:6803/165531318] [v2:127.0.0.1:6804/165531318,v1:127.0.0.1:6805/165531318] exists,up 680298c1-f1f0-4efb-ba2b-6125284012aa 2024-05-08T17:12:28.587 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-08T17:12:28.587 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-08T17:12:28.587 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-08T17:12:28.587 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-05-08T17:12:28.587 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 1 2024-05-08T17:12:28.587 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-05-08T17:12:28.587 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-05-08T17:12:28.587 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-05-08T17:12:28.588 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-05-08T17:12:28.588 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/1 2024-05-08T17:12:28.588 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=4457b0f1-028b-482b-8ee3-7ee05861e6cc --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-05-08T17:12:28.588 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-08T17:12:28.588 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-05-08T17:12:28.588 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-08T17:12:28.588 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/1' 2024-05-08T17:12:28.588 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/1/journal' 2024-05-08T17:12:28.588 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-05-08T17:12:28.588 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-05-08T17:12:28.588 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-05-08T17:12:28.590 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-05-08T17:12:28.590 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-08T17:12:28.590 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-08T17:12:28.591 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-08T17:12:28.591 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T17:12:28.591 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81110 2024-05-08T17:12:28.591 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.81110/$cluster-$name.asok' 2024-05-08T17:12:28.591 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.81110/$cluster-$name.asok' 2024-05-08T17:12:28.591 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-05-08T17:12:28.591 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-05-08T17:12:28.591 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-05-08T17:12:28.591 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-05-08T17:12:28.591 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-05-08T17:12:28.591 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-08T17:12:28.591 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-08T17:12:28.591 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-08T17:12:28.591 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-08T17:12:28.592 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-05-08T17:12:28.592 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-05-08T17:12:28.592 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/1 2024-05-08T17:12:28.592 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-05-08T17:12:28.592 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=bd039f30-dc86-46f2-9b57-85a7377a4f9b 2024-05-08T17:12:28.592 INFO:tasks.workunit.client.0.smithi172.stdout:add osd1 bd039f30-dc86-46f2-9b57-85a7377a4f9b 2024-05-08T17:12:28.593 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 bd039f30-dc86-46f2-9b57-85a7377a4f9b' 2024-05-08T17:12:28.593 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-05-08T17:12:28.613 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQB8sjtmqN3rJBAA7LkcTbu0k25BA9f8Rf0daA== 2024-05-08T17:12:28.613 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQB8sjtmqN3rJBAA7LkcTbu0k25BA9f8Rf0daA=="}' 2024-05-08T17:12:28.613 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new bd039f30-dc86-46f2-9b57-85a7377a4f9b -i td/test-erasure-eio/1/new.json 2024-05-08T17:12:29.076 INFO:tasks.workunit.client.0.smithi172.stdout:1 2024-05-08T17:12:29.093 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/1/new.json 2024-05-08T17:12:29.094 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=4457b0f1-028b-482b-8ee3-7ee05861e6cc --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81110/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQB8sjtmqN3rJBAA7LkcTbu0k25BA9f8Rf0daA== --osd-uuid bd039f30-dc86-46f2-9b57-85a7377a4f9b 2024-05-08T17:12:29.128 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:12:29.134+0000 7f2867c4e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T17:12:29.144 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:12:29.146+0000 7f2867c4e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T17:12:29.145 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:12:29.150+0000 7f2867c4e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T17:12:29.145 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:12:29.150+0000 7f2867c4e7c0 -1 bluestore(td/test-erasure-eio/1/block) _read_bdev_label failed to open td/test-erasure-eio/1/block: (2) No such file or directory 2024-05-08T17:12:29.145 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:12:29.150+0000 7f2867c4e7c0 -1 bluestore(td/test-erasure-eio/1/block) _read_bdev_label failed to open td/test-erasure-eio/1/block: (2) No such file or directory 2024-05-08T17:12:29.145 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:12:29.150+0000 7f2867c4e7c0 -1 bluestore(td/test-erasure-eio/1/block) _read_bdev_label failed to open td/test-erasure-eio/1/block: (2) No such file or directory 2024-05-08T17:12:29.243 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:12:29.246+0000 7f2867c4e7c0 -1 bluestore(td/test-erasure-eio/1) _read_fsid unparsable uuid 2024-05-08T17:12:35.297 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/1/keyring 2024-05-08T17:12:35.297 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-05-08T17:12:35.298 INFO:tasks.workunit.client.0.smithi172.stdout:adding osd1 key to auth repository 2024-05-08T17:12:35.298 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-05-08T17:12:35.299 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-05-08T17:12:35.866 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-05-08T17:12:35.867 INFO:tasks.workunit.client.0.smithi172.stdout:start osd.1 2024-05-08T17:12:35.867 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=4457b0f1-028b-482b-8ee3-7ee05861e6cc --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81110/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-05-08T17:12:35.867 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-05-08T17:12:35.867 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-05-08T17:12:35.867 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-05-08T17:12:35.892 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:12:35.898+0000 7f6222ec67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T17:12:35.894 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:12:35.898+0000 7f6222ec67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T17:12:35.895 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:12:35.902+0000 7f6222ec67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T17:12:36.328 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-05-08T17:12:36.329 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-08T17:12:36.329 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-05-08T17:12:36.329 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-08T17:12:36.329 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-08T17:12:36.329 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:12:36.329 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-08T17:12:36.329 INFO:tasks.workunit.client.0.smithi172.stdout:0 2024-05-08T17:12:36.330 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:12:36.330 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T17:12:36.453 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:12:36.458+0000 7f6222ec67c0 -1 Falling back to public interface 2024-05-08T17:12:36.794 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:12:37.796 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:12:37.796 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:12:37.796 INFO:tasks.workunit.client.0.smithi172.stdout:1 2024-05-08T17:12:37.797 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-08T17:12:37.797 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:12:37.797 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T17:12:38.198 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:12:38.934 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:12:38.938+0000 7f6222ec67c0 -1 osd.1 0 log_to_monitors true 2024-05-08T17:12:39.199 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:12:39.199 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:12:39.200 INFO:tasks.workunit.client.0.smithi172.stdout:2 2024-05-08T17:12:39.200 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-08T17:12:39.200 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:12:39.200 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T17:12:39.867 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:12:40.868 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:12:40.868 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:12:40.869 INFO:tasks.workunit.client.0.smithi172.stdout:3 2024-05-08T17:12:40.869 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-05-08T17:12:40.869 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:12:40.869 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T17:12:41.303 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:12:42.305 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:12:42.305 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:12:42.306 INFO:tasks.workunit.client.0.smithi172.stdout:4 2024-05-08T17:12:42.306 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-05-08T17:12:42.306 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:12:42.306 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T17:12:42.850 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:12:43.852 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:12:43.852 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:12:43.852 INFO:tasks.workunit.client.0.smithi172.stdout:5 2024-05-08T17:12:43.853 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-05-08T17:12:43.853 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:12:43.853 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T17:12:44.414 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:12:45.415 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:12:45.416 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:12:45.416 INFO:tasks.workunit.client.0.smithi172.stdout:6 2024-05-08T17:12:45.416 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-05-08T17:12:45.417 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:12:45.417 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T17:12:45.872 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:12:46.874 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:12:46.874 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:12:46.875 INFO:tasks.workunit.client.0.smithi172.stdout:7 2024-05-08T17:12:46.875 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-05-08T17:12:46.875 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:12:46.875 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T17:12:47.320 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:12:48.322 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:12:48.322 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:12:48.323 INFO:tasks.workunit.client.0.smithi172.stdout:8 2024-05-08T17:12:48.323 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-05-08T17:12:48.323 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:12:48.323 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T17:12:48.798 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:12:49.800 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:12:49.800 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:12:49.800 INFO:tasks.workunit.client.0.smithi172.stdout:9 2024-05-08T17:12:49.801 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-05-08T17:12:49.801 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:12:49.801 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T17:12:50.224 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:12:51.226 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:12:51.226 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:12:51.226 INFO:tasks.workunit.client.0.smithi172.stdout:10 2024-05-08T17:12:51.227 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-05-08T17:12:51.227 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:12:51.227 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T17:12:52.453 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:12:53.454 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:12:53.454 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:12:53.455 INFO:tasks.workunit.client.0.smithi172.stdout:11 2024-05-08T17:12:53.455 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-05-08T17:12:53.455 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:12:53.455 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T17:12:54.261 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:12:55.263 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:12:55.263 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:12:55.263 INFO:tasks.workunit.client.0.smithi172.stdout:12 2024-05-08T17:12:55.264 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-05-08T17:12:55.264 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:12:55.264 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T17:12:56.104 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:12:57.105 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:12:57.106 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:12:57.106 INFO:tasks.workunit.client.0.smithi172.stdout:13 2024-05-08T17:12:57.106 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-05-08T17:12:57.107 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:12:57.107 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T17:12:58.353 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:12:59.355 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:12:59.355 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:12:59.355 INFO:tasks.workunit.client.0.smithi172.stdout:14 2024-05-08T17:12:59.356 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-05-08T17:12:59.356 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:12:59.356 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T17:13:00.262 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:13:01.264 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:13:01.264 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:13:01.264 INFO:tasks.workunit.client.0.smithi172.stdout:15 2024-05-08T17:13:01.265 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-05-08T17:13:01.265 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:13:01.265 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T17:13:01.794 INFO:tasks.workunit.client.0.smithi172.stdout:osd.1 up in weight 1 up_from 13 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/4061311693,v1:127.0.0.1:6811/4061311693] [v2:127.0.0.1:6812/4061311693,v1:127.0.0.1:6813/4061311693] exists,up bd039f30-dc86-46f2-9b57-85a7377a4f9b 2024-05-08T17:13:01.798 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-08T17:13:01.798 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-08T17:13:01.798 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-08T17:13:01.798 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-05-08T17:13:01.798 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 2 2024-05-08T17:13:01.799 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-05-08T17:13:01.799 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-05-08T17:13:01.799 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-05-08T17:13:01.799 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-05-08T17:13:01.799 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/2 2024-05-08T17:13:01.799 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=4457b0f1-028b-482b-8ee3-7ee05861e6cc --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-05-08T17:13:01.799 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-08T17:13:01.799 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-05-08T17:13:01.799 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-08T17:13:01.799 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/2' 2024-05-08T17:13:01.799 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/2/journal' 2024-05-08T17:13:01.799 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-05-08T17:13:01.799 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-05-08T17:13:01.800 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-05-08T17:13:01.800 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-05-08T17:13:01.800 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-08T17:13:01.800 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-08T17:13:01.800 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-08T17:13:01.800 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T17:13:01.800 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81110 2024-05-08T17:13:01.800 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.81110/$cluster-$name.asok' 2024-05-08T17:13:01.800 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.81110/$cluster-$name.asok' 2024-05-08T17:13:01.800 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-05-08T17:13:01.800 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-05-08T17:13:01.800 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-05-08T17:13:01.800 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-05-08T17:13:01.800 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-05-08T17:13:01.800 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-08T17:13:01.801 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-08T17:13:01.801 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-08T17:13:01.801 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-08T17:13:01.801 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-05-08T17:13:01.801 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-05-08T17:13:01.801 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/2 2024-05-08T17:13:01.827 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-05-08T17:13:01.828 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=5a9950b0-f3fa-42d4-8f6b-9780b2f59359 2024-05-08T17:13:01.828 INFO:tasks.workunit.client.0.smithi172.stdout:add osd2 5a9950b0-f3fa-42d4-8f6b-9780b2f59359 2024-05-08T17:13:01.829 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 5a9950b0-f3fa-42d4-8f6b-9780b2f59359' 2024-05-08T17:13:01.829 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-05-08T17:13:01.852 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCdsjtmXCwhMxAAdfP1L+yf0y94cPOxE/apng== 2024-05-08T17:13:01.852 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCdsjtmXCwhMxAAdfP1L+yf0y94cPOxE/apng=="}' 2024-05-08T17:13:01.852 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 5a9950b0-f3fa-42d4-8f6b-9780b2f59359 -i td/test-erasure-eio/2/new.json 2024-05-08T17:13:02.644 INFO:tasks.workunit.client.0.smithi172.stdout:2 2024-05-08T17:13:02.660 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/2/new.json 2024-05-08T17:13:02.661 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=4457b0f1-028b-482b-8ee3-7ee05861e6cc --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81110/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCdsjtmXCwhMxAAdfP1L+yf0y94cPOxE/apng== --osd-uuid 5a9950b0-f3fa-42d4-8f6b-9780b2f59359 2024-05-08T17:13:02.695 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:13:02.698+0000 7f32aa7747c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T17:13:02.702 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:13:02.706+0000 7f32aa7747c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T17:13:02.703 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:13:02.706+0000 7f32aa7747c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T17:13:02.703 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:13:02.706+0000 7f32aa7747c0 -1 bluestore(td/test-erasure-eio/2/block) _read_bdev_label failed to open td/test-erasure-eio/2/block: (2) No such file or directory 2024-05-08T17:13:02.703 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:13:02.706+0000 7f32aa7747c0 -1 bluestore(td/test-erasure-eio/2/block) _read_bdev_label failed to open td/test-erasure-eio/2/block: (2) No such file or directory 2024-05-08T17:13:02.703 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:13:02.706+0000 7f32aa7747c0 -1 bluestore(td/test-erasure-eio/2/block) _read_bdev_label failed to open td/test-erasure-eio/2/block: (2) No such file or directory 2024-05-08T17:13:02.747 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:13:02.750+0000 7f32aa7747c0 -1 bluestore(td/test-erasure-eio/2) _read_fsid unparsable uuid 2024-05-08T17:13:08.869 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/2/keyring 2024-05-08T17:13:08.869 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-05-08T17:13:08.870 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-05-08T17:13:08.870 INFO:tasks.workunit.client.0.smithi172.stdout:adding osd2 key to auth repository 2024-05-08T17:13:08.870 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-05-08T17:13:09.424 INFO:tasks.workunit.client.0.smithi172.stdout:start osd.2 2024-05-08T17:13:09.425 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-05-08T17:13:09.425 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=4457b0f1-028b-482b-8ee3-7ee05861e6cc --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81110/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-05-08T17:13:09.425 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-05-08T17:13:09.425 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-05-08T17:13:09.425 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-05-08T17:13:09.455 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:13:09.457+0000 7fc1d0e5f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T17:13:09.490 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:13:09.493+0000 7fc1d0e5f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T17:13:09.492 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:13:09.497+0000 7fc1d0e5f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T17:13:09.836 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-05-08T17:13:09.836 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-08T17:13:09.836 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-05-08T17:13:09.836 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-08T17:13:09.836 INFO:tasks.workunit.client.0.smithi172.stdout:0 2024-05-08T17:13:09.837 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-08T17:13:09.837 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:13:09.837 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-08T17:13:09.837 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:13:09.837 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T17:13:10.053 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:13:10.057+0000 7fc1d0e5f7c0 -1 Falling back to public interface 2024-05-08T17:13:10.261 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:13:11.267 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:13:11.267 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:13:11.267 INFO:tasks.workunit.client.0.smithi172.stdout:1 2024-05-08T17:13:11.268 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-08T17:13:11.268 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:13:11.268 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T17:13:11.687 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:13:12.561 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:13:12.565+0000 7fc1d0e5f7c0 -1 osd.2 0 log_to_monitors true 2024-05-08T17:13:12.689 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:13:12.689 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:13:12.689 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-08T17:13:12.690 INFO:tasks.workunit.client.0.smithi172.stdout:2 2024-05-08T17:13:12.690 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:13:12.690 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T17:13:13.282 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:13:14.283 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:13:14.283 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:13:14.284 INFO:tasks.workunit.client.0.smithi172.stdout:3 2024-05-08T17:13:14.284 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-05-08T17:13:14.284 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:13:14.284 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T17:13:14.780 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:13:15.782 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:13:15.782 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:13:15.782 INFO:tasks.workunit.client.0.smithi172.stdout:4 2024-05-08T17:13:15.783 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-05-08T17:13:15.783 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:13:15.784 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T17:13:16.258 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:13:17.260 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:13:17.260 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:13:17.260 INFO:tasks.workunit.client.0.smithi172.stdout:5 2024-05-08T17:13:17.261 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-05-08T17:13:17.261 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:13:17.261 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T17:13:17.674 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:13:18.676 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:13:18.676 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:13:18.677 INFO:tasks.workunit.client.0.smithi172.stdout:6 2024-05-08T17:13:18.677 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-05-08T17:13:18.677 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:13:18.677 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T17:13:19.094 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:13:20.095 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:13:20.096 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:13:20.096 INFO:tasks.workunit.client.0.smithi172.stdout:7 2024-05-08T17:13:20.096 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-05-08T17:13:20.096 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:13:20.096 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T17:13:20.597 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:13:21.599 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:13:21.599 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:13:21.599 INFO:tasks.workunit.client.0.smithi172.stdout:8 2024-05-08T17:13:21.600 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-05-08T17:13:21.600 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:13:21.600 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T17:13:22.027 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:13:23.029 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:13:23.029 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:13:23.029 INFO:tasks.workunit.client.0.smithi172.stdout:9 2024-05-08T17:13:23.030 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-05-08T17:13:23.030 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:13:23.030 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T17:13:23.603 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:13:24.605 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:13:24.605 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:13:24.605 INFO:tasks.workunit.client.0.smithi172.stdout:10 2024-05-08T17:13:24.605 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-05-08T17:13:24.606 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:13:24.606 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T17:13:25.048 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:13:26.050 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:13:26.050 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:13:26.050 INFO:tasks.workunit.client.0.smithi172.stdout:11 2024-05-08T17:13:26.051 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-05-08T17:13:26.051 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:13:26.051 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T17:13:26.845 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:13:27.846 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:13:27.847 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:13:27.847 INFO:tasks.workunit.client.0.smithi172.stdout:12 2024-05-08T17:13:27.847 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-05-08T17:13:27.848 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:13:27.848 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T17:13:28.645 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:13:29.646 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:13:29.646 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:13:29.646 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-05-08T17:13:29.647 INFO:tasks.workunit.client.0.smithi172.stdout:13 2024-05-08T17:13:29.647 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:13:29.647 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T17:13:30.632 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:13:31.634 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:13:31.634 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:13:31.635 INFO:tasks.workunit.client.0.smithi172.stdout:14 2024-05-08T17:13:31.635 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-05-08T17:13:31.635 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:13:31.635 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T17:13:32.604 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:13:33.606 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:13:33.606 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:13:33.606 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-05-08T17:13:33.606 INFO:tasks.workunit.client.0.smithi172.stdout:15 2024-05-08T17:13:33.606 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:13:33.607 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T17:13:34.518 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:13:35.519 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:13:35.519 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:13:35.520 INFO:tasks.workunit.client.0.smithi172.stdout:16 2024-05-08T17:13:35.520 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-05-08T17:13:35.520 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:13:35.520 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T17:13:36.101 INFO:tasks.workunit.client.0.smithi172.stdout:osd.2 up in weight 1 up_from 19 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/222335594,v1:127.0.0.1:6819/222335594] [v2:127.0.0.1:6820/222335594,v1:127.0.0.1:6821/222335594] exists,up 5a9950b0-f3fa-42d4-8f6b-9780b2f59359 2024-05-08T17:13:36.103 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-08T17:13:36.103 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-08T17:13:36.103 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-08T17:13:36.103 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-05-08T17:13:36.103 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 3 2024-05-08T17:13:36.104 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-05-08T17:13:36.104 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-05-08T17:13:36.104 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2024-05-08T17:13:36.104 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-05-08T17:13:36.104 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/3 2024-05-08T17:13:36.104 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=4457b0f1-028b-482b-8ee3-7ee05861e6cc --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-05-08T17:13:36.104 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-08T17:13:36.104 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-05-08T17:13:36.104 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-08T17:13:36.104 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/3' 2024-05-08T17:13:36.104 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/3/journal' 2024-05-08T17:13:36.104 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-05-08T17:13:36.104 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-05-08T17:13:36.104 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-05-08T17:13:36.105 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-05-08T17:13:36.105 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-08T17:13:36.105 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-08T17:13:36.105 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-08T17:13:36.105 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T17:13:36.105 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81110 2024-05-08T17:13:36.106 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.81110/$cluster-$name.asok' 2024-05-08T17:13:36.107 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.81110/$cluster-$name.asok' 2024-05-08T17:13:36.107 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-05-08T17:13:36.107 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-05-08T17:13:36.107 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-05-08T17:13:36.107 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-05-08T17:13:36.107 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-05-08T17:13:36.107 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-08T17:13:36.107 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-08T17:13:36.107 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-08T17:13:36.107 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-08T17:13:36.107 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-05-08T17:13:36.107 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-05-08T17:13:36.107 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/3 2024-05-08T17:13:36.107 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-05-08T17:13:36.108 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=53ff78af-91ec-437b-80b2-7a5d0b6789f2 2024-05-08T17:13:36.108 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 53ff78af-91ec-437b-80b2-7a5d0b6789f2' 2024-05-08T17:13:36.108 INFO:tasks.workunit.client.0.smithi172.stdout:add osd3 53ff78af-91ec-437b-80b2-7a5d0b6789f2 2024-05-08T17:13:36.108 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-05-08T17:13:36.130 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDAsjtmJuMjCBAAjI6gkziVIVq/Vmqn6qIk7w== 2024-05-08T17:13:36.130 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDAsjtmJuMjCBAAjI6gkziVIVq/Vmqn6qIk7w=="}' 2024-05-08T17:13:36.130 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 53ff78af-91ec-437b-80b2-7a5d0b6789f2 -i td/test-erasure-eio/3/new.json 2024-05-08T17:13:36.669 INFO:tasks.workunit.client.0.smithi172.stdout:3 2024-05-08T17:13:36.686 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/3/new.json 2024-05-08T17:13:36.687 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=4457b0f1-028b-482b-8ee3-7ee05861e6cc --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81110/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDAsjtmJuMjCBAAjI6gkziVIVq/Vmqn6qIk7w== --osd-uuid 53ff78af-91ec-437b-80b2-7a5d0b6789f2 2024-05-08T17:13:36.720 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:13:36.725+0000 7f40c04367c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T17:13:36.722 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:13:36.725+0000 7f40c04367c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T17:13:36.724 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:13:36.729+0000 7f40c04367c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T17:13:36.724 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:13:36.729+0000 7f40c04367c0 -1 bluestore(td/test-erasure-eio/3/block) _read_bdev_label failed to open td/test-erasure-eio/3/block: (2) No such file or directory 2024-05-08T17:13:36.724 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:13:36.729+0000 7f40c04367c0 -1 bluestore(td/test-erasure-eio/3/block) _read_bdev_label failed to open td/test-erasure-eio/3/block: (2) No such file or directory 2024-05-08T17:13:36.724 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:13:36.729+0000 7f40c04367c0 -1 bluestore(td/test-erasure-eio/3/block) _read_bdev_label failed to open td/test-erasure-eio/3/block: (2) No such file or directory 2024-05-08T17:13:36.826 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:13:36.829+0000 7f40c04367c0 -1 bluestore(td/test-erasure-eio/3) _read_fsid unparsable uuid 2024-05-08T17:13:42.913 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/3/keyring 2024-05-08T17:13:42.913 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-05-08T17:13:42.914 INFO:tasks.workunit.client.0.smithi172.stdout:adding osd3 key to auth repository 2024-05-08T17:13:42.915 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2024-05-08T17:13:42.915 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-05-08T17:13:43.480 INFO:tasks.workunit.client.0.smithi172.stdout:start osd.3 2024-05-08T17:13:43.481 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2024-05-08T17:13:43.481 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=4457b0f1-028b-482b-8ee3-7ee05861e6cc --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81110/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-05-08T17:13:43.481 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-05-08T17:13:43.481 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-05-08T17:13:43.481 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-05-08T17:13:43.511 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:13:43.513+0000 7f6ca50857c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T17:13:43.525 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:13:43.529+0000 7f6ca50857c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T17:13:43.526 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:13:43.529+0000 7f6ca50857c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T17:13:43.921 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2024-05-08T17:13:43.922 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-08T17:13:43.922 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-05-08T17:13:43.922 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-08T17:13:43.922 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-08T17:13:43.922 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:13:43.922 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-08T17:13:43.922 INFO:tasks.workunit.client.0.smithi172.stdout:0 2024-05-08T17:13:43.923 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:13:43.923 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T17:13:44.089 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:13:44.093+0000 7f6ca50857c0 -1 Falling back to public interface 2024-05-08T17:13:44.346 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:13:45.348 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:13:45.348 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:13:45.349 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-08T17:13:45.349 INFO:tasks.workunit.client.0.smithi172.stdout:1 2024-05-08T17:13:45.349 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:13:45.349 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T17:13:45.750 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:13:46.657 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:13:46.660+0000 7f6ca50857c0 -1 osd.3 0 log_to_monitors true 2024-05-08T17:13:46.752 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:13:46.752 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:13:46.752 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-08T17:13:46.752 INFO:tasks.workunit.client.0.smithi172.stdout:2 2024-05-08T17:13:46.753 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:13:46.753 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T17:13:47.466 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:13:48.467 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:13:48.467 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:13:48.468 INFO:tasks.workunit.client.0.smithi172.stdout:3 2024-05-08T17:13:48.468 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-05-08T17:13:48.468 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:13:48.468 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T17:13:49.154 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:13:50.155 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:13:50.155 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:13:50.156 INFO:tasks.workunit.client.0.smithi172.stdout:4 2024-05-08T17:13:50.156 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-05-08T17:13:50.156 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:13:50.156 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T17:13:50.579 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:13:51.581 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:13:51.581 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:13:51.581 INFO:tasks.workunit.client.0.smithi172.stdout:5 2024-05-08T17:13:51.582 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-05-08T17:13:51.582 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:13:51.582 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T17:13:51.996 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:13:52.998 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:13:52.998 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:13:52.999 INFO:tasks.workunit.client.0.smithi172.stdout:6 2024-05-08T17:13:52.999 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-05-08T17:13:53.000 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:13:53.000 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T17:13:53.506 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:13:54.506 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:13:54.507 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:13:54.507 INFO:tasks.workunit.client.0.smithi172.stdout:7 2024-05-08T17:13:54.507 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-05-08T17:13:54.508 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:13:54.508 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T17:13:54.927 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:13:55.929 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:13:55.929 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:13:55.929 INFO:tasks.workunit.client.0.smithi172.stdout:8 2024-05-08T17:13:55.930 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-05-08T17:13:55.930 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:13:55.930 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T17:13:56.475 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:13:57.477 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:13:57.478 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:13:57.478 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-05-08T17:13:57.478 INFO:tasks.workunit.client.0.smithi172.stdout:9 2024-05-08T17:13:57.478 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:13:57.478 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T17:13:58.037 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:13:59.039 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:13:59.039 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:13:59.039 INFO:tasks.workunit.client.0.smithi172.stdout:10 2024-05-08T17:13:59.040 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-05-08T17:13:59.040 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:13:59.040 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T17:13:59.778 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:14:00.780 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:14:00.780 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:14:00.780 INFO:tasks.workunit.client.0.smithi172.stdout:11 2024-05-08T17:14:00.780 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-05-08T17:14:00.781 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:14:00.781 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T17:14:01.532 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:14:02.534 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:14:02.534 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:14:02.534 INFO:tasks.workunit.client.0.smithi172.stdout:12 2024-05-08T17:14:02.535 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-05-08T17:14:02.535 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:14:02.535 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T17:14:03.278 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:14:04.280 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:14:04.281 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:14:04.281 INFO:tasks.workunit.client.0.smithi172.stdout:13 2024-05-08T17:14:04.281 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-05-08T17:14:04.282 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:14:04.282 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T17:14:05.066 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:14:06.068 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:14:06.068 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:14:06.068 INFO:tasks.workunit.client.0.smithi172.stdout:14 2024-05-08T17:14:06.069 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-05-08T17:14:06.069 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:14:06.069 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T17:14:07.114 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:14:08.116 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:14:08.117 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:14:08.117 INFO:tasks.workunit.client.0.smithi172.stdout:15 2024-05-08T17:14:08.117 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-05-08T17:14:08.117 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:14:08.117 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T17:14:09.061 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:14:10.063 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:14:10.063 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:14:10.063 INFO:tasks.workunit.client.0.smithi172.stdout:16 2024-05-08T17:14:10.064 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-05-08T17:14:10.064 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:14:10.064 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T17:14:10.685 INFO:tasks.workunit.client.0.smithi172.stdout:osd.3 up in weight 1 up_from 27 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/1151300699,v1:127.0.0.1:6827/1151300699] [v2:127.0.0.1:6828/1151300699,v1:127.0.0.1:6829/1151300699] exists,up 53ff78af-91ec-437b-80b2-7a5d0b6789f2 2024-05-08T17:14:10.686 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-08T17:14:10.686 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-08T17:14:10.686 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-08T17:14:10.686 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: get_asok_path osd.0 2024-05-08T17:14:10.686 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-05-08T17:14:10.686 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-05-08T17:14:10.687 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-08T17:14:10.687 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T17:14:10.687 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81110 2024-05-08T17:14:10.688 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.81110/ceph-osd.0.asok 2024-05-08T17:14:10.688 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: CEPH_ARGS= 2024-05-08T17:14:10.688 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: ceph --admin-daemon /tmp/ceph-asok.81110/ceph-osd.0.asok log flush 2024-05-08T17:14:10.799 INFO:tasks.workunit.client.0.smithi172.stdout:{} 2024-05-08T17:14:10.812 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:56: setup_osds: grep 'load: jerasure.*lrc' td/test-erasure-eio/osd.0.log 2024-05-08T17:14:10.818 INFO:tasks.workunit.client.0.smithi172.stdout:2024-05-08T17:12:03.163+0000 7fdfec5687c0 0 load: jerasure load: lrc 2024-05-08T17:14:10.819 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:285: TEST_rados_get_subread_missing: local poolname=pool-jerasure 2024-05-08T17:14:10.819 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:286: TEST_rados_get_subread_missing: create_erasure_coded_pool pool-jerasure 2 1 2024-05-08T17:14:10.819 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:67: create_erasure_coded_pool: local poolname=pool-jerasure 2024-05-08T17:14:10.819 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:68: create_erasure_coded_pool: shift 2024-05-08T17:14:10.819 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:69: create_erasure_coded_pool: local k=2 2024-05-08T17:14:10.819 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:70: create_erasure_coded_pool: shift 2024-05-08T17:14:10.820 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:71: create_erasure_coded_pool: local m=1 2024-05-08T17:14:10.820 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:72: create_erasure_coded_pool: shift 2024-05-08T17:14:10.820 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:74: create_erasure_coded_pool: ceph osd erasure-code-profile set myprofile plugin=jerasure k=2 m=1 crush-failure-domain=osd 2024-05-08T17:14:11.811 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:78: create_erasure_coded_pool: create_pool pool-jerasure 1 1 erasure myprofile 2024-05-08T17:14:11.811 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create pool-jerasure 1 1 erasure myprofile 2024-05-08T17:14:12.494 INFO:tasks.workunit.client.0.smithi172.stderr:pool 'pool-jerasure' already exists 2024-05-08T17:14:12.512 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-05-08T17:14:13.514 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:80: create_erasure_coded_pool: wait_for_clean 2024-05-08T17:14:13.514 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-05-08T17:14:13.514 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-05-08T17:14:13.514 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-05-08T17:14:13.515 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-05-08T17:14:13.516 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-05-08T17:14:13.516 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-05-08T17:14:13.517 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-05-08T17:14:13.517 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-05-08T17:14:13.517 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-05-08T17:14:13.648 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-05-08T17:14:13.648 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-05-08T17:14:13.648 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-05-08T17:14:13.648 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-05-08T17:14:13.648 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-05-08T17:14:13.648 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-05-08T17:14:14.083 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-05-08T17:14:14.083 INFO:tasks.workunit.client.0.smithi172.stderr:1 2024-05-08T17:14:14.084 INFO:tasks.workunit.client.0.smithi172.stderr:2 2024-05-08T17:14:14.084 INFO:tasks.workunit.client.0.smithi172.stderr:3' 2024-05-08T17:14:14.084 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-05-08T17:14:14.084 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T17:14:14.084 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-05-08T17:14:14.281 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=25769803803 2024-05-08T17:14:14.282 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 25769803803 2024-05-08T17:14:14.282 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803803' 2024-05-08T17:14:14.282 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T17:14:14.282 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-05-08T17:14:14.435 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=55834574869 2024-05-08T17:14:14.435 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 55834574869 2024-05-08T17:14:14.435 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803803 1-55834574869' 2024-05-08T17:14:14.435 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T17:14:14.435 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-05-08T17:14:14.587 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=81604378637 2024-05-08T17:14:14.588 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 81604378637 2024-05-08T17:14:14.588 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803803 1-55834574869 2-81604378637' 2024-05-08T17:14:14.588 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T17:14:14.588 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-05-08T17:14:14.747 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=115964116999 2024-05-08T17:14:14.748 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 115964116999 2024-05-08T17:14:14.748 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803803 1-55834574869 2-81604378637 3-115964116999' 2024-05-08T17:14:14.748 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-08T17:14:14.748 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-25769803803 2024-05-08T17:14:14.748 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-08T17:14:14.749 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-05-08T17:14:14.750 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-25769803803 2024-05-08T17:14:14.750 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-08T17:14:14.752 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=25769803803 2024-05-08T17:14:14.752 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 25769803803' 2024-05-08T17:14:14.752 INFO:tasks.workunit.client.0.smithi172.stdout:waiting osd.0 seq 25769803803 2024-05-08T17:14:14.752 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-05-08T17:14:15.163 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 25769803803 -lt 25769803803 2024-05-08T17:14:15.163 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-08T17:14:15.164 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-08T17:14:15.164 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-55834574869 2024-05-08T17:14:15.165 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-05-08T17:14:15.166 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-55834574869 2024-05-08T17:14:15.166 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-08T17:14:15.167 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=55834574869 2024-05-08T17:14:15.167 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 55834574869' 2024-05-08T17:14:15.168 INFO:tasks.workunit.client.0.smithi172.stdout:waiting osd.1 seq 55834574869 2024-05-08T17:14:15.168 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-05-08T17:14:15.635 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 55834574869 -lt 55834574869 2024-05-08T17:14:15.635 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-08T17:14:15.636 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-81604378637 2024-05-08T17:14:15.636 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-08T17:14:15.637 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-05-08T17:14:15.638 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-81604378637 2024-05-08T17:14:15.638 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-08T17:14:15.639 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=81604378637 2024-05-08T17:14:15.639 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 81604378637' 2024-05-08T17:14:15.640 INFO:tasks.workunit.client.0.smithi172.stdout:waiting osd.2 seq 81604378637 2024-05-08T17:14:15.640 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-05-08T17:14:16.095 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 81604378637 -lt 81604378637 2024-05-08T17:14:16.096 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-08T17:14:16.096 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-115964116999 2024-05-08T17:14:16.096 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-08T17:14:16.097 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-05-08T17:14:16.098 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-115964116999 2024-05-08T17:14:16.098 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-08T17:14:16.100 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=115964116999 2024-05-08T17:14:16.100 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 115964116999' 2024-05-08T17:14:16.100 INFO:tasks.workunit.client.0.smithi172.stdout:waiting osd.3 seq 115964116999 2024-05-08T17:14:16.100 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-05-08T17:14:16.546 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 115964116999 -lt 115964116999 2024-05-08T17:14:16.547 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-05-08T17:14:16.548 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-05-08T17:14:16.548 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-05-08T17:14:17.096 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-05-08T17:14:17.097 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-05-08T17:14:17.097 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-05-08T17:14:17.097 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-05-08T17:14:17.097 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-05-08T17:14:17.097 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-05-08T17:14:17.097 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-05-08T17:14:17.097 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-05-08T17:14:17.478 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-05-08T17:14:17.479 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-05-08T17:14:17.479 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-05-08T17:14:17.479 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-05-08T17:14:17.992 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-05-08T17:14:17.993 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-05-08T17:14:17.993 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-05-08T17:14:17.993 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:288: TEST_rados_get_subread_missing: local shard_id=1 2024-05-08T17:14:17.993 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:289: TEST_rados_get_subread_missing: rados_put_get_data remove td/test-erasure-eio 1 2024-05-08T17:14:17.993 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:147: rados_put_get_data: local inject=remove 2024-05-08T17:14:17.993 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:148: rados_put_get_data: shift 2024-05-08T17:14:17.993 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:149: rados_put_get_data: local dir=td/test-erasure-eio 2024-05-08T17:14:17.993 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:150: rados_put_get_data: shift 2024-05-08T17:14:17.993 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:151: rados_put_get_data: local shard_id=1 2024-05-08T17:14:17.994 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:152: rados_put_get_data: shift 2024-05-08T17:14:17.994 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:153: rados_put_get_data: local arg= 2024-05-08T17:14:17.994 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:157: rados_put_get_data: local poolname=pool-jerasure 2024-05-08T17:14:17.994 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:158: rados_put_get_data: local objname=obj-remove-81110-1 2024-05-08T17:14:17.994 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:159: rados_put_get_data: rados_put td/test-erasure-eio pool-jerasure obj-remove-81110-1 2024-05-08T17:14:17.994 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:90: rados_put: local dir=td/test-erasure-eio 2024-05-08T17:14:17.994 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:91: rados_put: local poolname=pool-jerasure 2024-05-08T17:14:17.994 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:92: rados_put: local objname=obj-remove-81110-1 2024-05-08T17:14:17.994 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-05-08T17:14:17.994 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 AAA 2024-05-08T17:14:17.994 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-05-08T17:14:17.995 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 BBB 2024-05-08T17:14:17.995 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-05-08T17:14:17.995 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 CCCC 2024-05-08T17:14:17.995 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-05-08T17:14:17.995 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 DDDD 2024-05-08T17:14:17.995 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:100: rados_put: rados --pool pool-jerasure put obj-remove-81110-1 td/test-erasure-eio/ORIGINAL 2024-05-08T17:14:18.177 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:160: rados_put_get_data: inject_remove ec data pool-jerasure obj-remove-81110-1 td/test-erasure-eio 1 2024-05-08T17:14:18.177 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:127: inject_remove: local pooltype=ec 2024-05-08T17:14:18.177 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:128: inject_remove: shift 2024-05-08T17:14:18.177 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:129: inject_remove: local which=data 2024-05-08T17:14:18.177 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:130: inject_remove: shift 2024-05-08T17:14:18.177 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:131: inject_remove: local poolname=pool-jerasure 2024-05-08T17:14:18.177 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:132: inject_remove: shift 2024-05-08T17:14:18.177 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:133: inject_remove: local objname=obj-remove-81110-1 2024-05-08T17:14:18.177 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:134: inject_remove: shift 2024-05-08T17:14:18.177 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:135: inject_remove: local dir=td/test-erasure-eio 2024-05-08T17:14:18.178 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:136: inject_remove: shift 2024-05-08T17:14:18.178 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:137: inject_remove: local shard_id=1 2024-05-08T17:14:18.178 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:138: inject_remove: shift 2024-05-08T17:14:18.178 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:140: inject_remove: get_osds pool-jerasure obj-remove-81110-1 2024-05-08T17:14:18.178 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2024-05-08T17:14:18.178 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-remove-81110-1 2024-05-08T17:14:18.178 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure obj-remove-81110-1 2024-05-08T17:14:18.179 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-05-08T17:14:18.645 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-05-08T17:14:18.645 INFO:tasks.workunit.client.0.smithi172.stderr:1 2024-05-08T17:14:18.645 INFO:tasks.workunit.client.0.smithi172.stderr:2' 2024-05-08T17:14:18.645 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 2 2024-05-08T17:14:18.647 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:140: inject_remove: initial_osds=('3' '1' '2') 2024-05-08T17:14:18.647 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:140: inject_remove: local -a initial_osds 2024-05-08T17:14:18.647 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:141: inject_remove: local osd_id=1 2024-05-08T17:14:18.647 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:142: inject_remove: objectstore_tool td/test-erasure-eio 1 obj-remove-81110-1 remove 2024-05-08T17:14:18.647 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/test-erasure-eio 2024-05-08T17:14:18.647 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-05-08T17:14:18.647 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-05-08T17:14:18.648 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-05-08T17:14:18.648 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/test-erasure-eio 1 obj-remove-81110-1 remove 2024-05-08T17:14:18.648 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/test-erasure-eio 2024-05-08T17:14:18.648 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-05-08T17:14:18.648 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-05-08T17:14:18.648 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-05-08T17:14:18.648 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/test-erasure-eio TERM osd.1 2024-05-08T17:14:18.648 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-08T17:14:18.648 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-08T17:14:18.648 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-08T17:14:18.648 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-08T17:14:18.648 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-08T17:14:18.957 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-08T17:14:18.957 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/test-erasure-eio 1 obj-remove-81110-1 remove 2024-05-08T17:14:18.957 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/test-erasure-eio 2024-05-08T17:14:18.957 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-05-08T17:14:18.957 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-05-08T17:14:18.958 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-05-08T17:14:18.958 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/test-erasure-eio/1 2024-05-08T17:14:18.958 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/test-erasure-eio/1 obj-remove-81110-1 remove 2024-05-08T17:14:21.137 INFO:tasks.workunit.client.0.smithi172.stdout:remove 1#2:d6f72261:::obj-remove-81110-1:head# 2024-05-08T17:14:21.674 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/test-erasure-eio 1 2024-05-08T17:14:21.675 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/test-erasure-eio 2024-05-08T17:14:21.675 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-05-08T17:14:21.675 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-05-08T17:14:21.675 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-05-08T17:14:21.675 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/test-erasure-eio/1 2024-05-08T17:14:21.675 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=4457b0f1-028b-482b-8ee3-7ee05861e6cc --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-05-08T17:14:21.675 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-08T17:14:21.675 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-05-08T17:14:21.675 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-08T17:14:21.675 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/1' 2024-05-08T17:14:21.675 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/1/journal' 2024-05-08T17:14:21.675 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-05-08T17:14:21.676 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-05-08T17:14:21.676 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-05-08T17:14:21.676 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-05-08T17:14:21.676 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-08T17:14:21.676 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-08T17:14:21.678 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-08T17:14:21.678 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T17:14:21.678 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81110 2024-05-08T17:14:21.678 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.81110/$cluster-$name.asok' 2024-05-08T17:14:21.678 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.81110/$cluster-$name.asok' 2024-05-08T17:14:21.678 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-05-08T17:14:21.678 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-05-08T17:14:21.678 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-05-08T17:14:21.678 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-08T17:14:21.678 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-08T17:14:21.678 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-08T17:14:21.678 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-08T17:14:21.679 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-05-08T17:14:21.679 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-05-08T17:14:21.679 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/test-erasure-eio/1 2024-05-08T17:14:21.679 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-05-08T17:14:21.679 INFO:tasks.workunit.client.0.smithi172.stderr:start osd.1 2024-05-08T17:14:21.679 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=4457b0f1-028b-482b-8ee3-7ee05861e6cc --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81110/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-05-08T17:14:21.680 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/test-erasure-eio/1/whoami 2024-05-08T17:14:21.680 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-05-08T17:14:21.680 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-05-08T17:14:21.680 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-05-08T17:14:21.680 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-05-08T17:14:21.712 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:14:21.712+0000 7fb9a86c77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T17:14:21.715 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:14:21.716+0000 7fb9a86c77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T17:14:21.717 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:14:21.720+0000 7fb9a86c77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T17:14:22.135 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-05-08T17:14:22.135 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-08T17:14:22.135 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-05-08T17:14:22.135 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-08T17:14:22.135 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-08T17:14:22.135 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:14:22.136 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-08T17:14:22.136 INFO:tasks.workunit.client.0.smithi172.stderr:0 2024-05-08T17:14:22.136 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T17:14:22.136 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:14:22.285 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:14:22.288+0000 7fb9a86c77c0 -1 Falling back to public interface 2024-05-08T17:14:22.645 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:14:23.647 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:14:23.647 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:14:23.647 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-08T17:14:23.647 INFO:tasks.workunit.client.0.smithi172.stderr:1 2024-05-08T17:14:23.648 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:14:23.648 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T17:14:24.066 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:14:24.781 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:14:24.784+0000 7fb9a86c77c0 -1 osd.1 36 log_to_monitors true 2024-05-08T17:14:25.068 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:14:25.068 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:14:25.068 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-08T17:14:25.069 INFO:tasks.workunit.client.0.smithi172.stderr:2 2024-05-08T17:14:25.069 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:14:25.069 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T17:14:25.829 INFO:tasks.workunit.client.0.smithi172.stderr:osd.1 up in weight 1 up_from 41 up_thru 41 down_at 37 last_clean_interval [13,36) [v2:127.0.0.1:6810/1682298543,v1:127.0.0.1:6811/1682298543] [v2:127.0.0.1:6812/1682298543,v1:127.0.0.1:6813/1682298543] exists,up bd039f30-dc86-46f2-9b57-85a7377a4f9b 2024-05-08T17:14:25.829 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-08T17:14:25.829 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-08T17:14:25.829 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-08T17:14:25.829 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-05-08T17:14:25.829 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-05-08T17:14:25.829 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-05-08T17:14:25.829 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-05-08T17:14:25.830 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-05-08T17:14:25.830 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-05-08T17:14:25.830 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-05-08T17:14:25.830 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-05-08T17:14:25.831 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-05-08T17:14:25.831 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-05-08T17:14:25.941 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-05-08T17:14:25.942 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-05-08T17:14:25.942 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-05-08T17:14:25.942 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-05-08T17:14:25.942 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-05-08T17:14:25.942 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-05-08T17:14:26.401 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-05-08T17:14:26.401 INFO:tasks.workunit.client.0.smithi172.stderr:1 2024-05-08T17:14:26.401 INFO:tasks.workunit.client.0.smithi172.stderr:2 2024-05-08T17:14:26.401 INFO:tasks.workunit.client.0.smithi172.stderr:3' 2024-05-08T17:14:26.401 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-05-08T17:14:26.401 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T17:14:26.402 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-05-08T17:14:26.559 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=25769803807 2024-05-08T17:14:26.559 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 25769803807 2024-05-08T17:14:26.560 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803807' 2024-05-08T17:14:26.560 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T17:14:26.560 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-05-08T17:14:26.722 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=176093659139 2024-05-08T17:14:26.722 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 176093659139 2024-05-08T17:14:26.722 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803807 1-176093659139' 2024-05-08T17:14:26.722 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T17:14:26.723 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-05-08T17:14:26.872 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=81604378642 2024-05-08T17:14:26.872 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 81604378642 2024-05-08T17:14:26.872 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803807 1-176093659139 2-81604378642' 2024-05-08T17:14:26.872 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T17:14:26.872 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-05-08T17:14:27.021 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=115964117004 2024-05-08T17:14:27.021 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 115964117004 2024-05-08T17:14:27.021 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803807 1-176093659139 2-81604378642 3-115964117004' 2024-05-08T17:14:27.021 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-08T17:14:27.021 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-25769803807 2024-05-08T17:14:27.021 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-08T17:14:27.022 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-05-08T17:14:27.023 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-25769803807 2024-05-08T17:14:27.023 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-08T17:14:27.024 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=25769803807 2024-05-08T17:14:27.024 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 25769803807' 2024-05-08T17:14:27.024 INFO:tasks.workunit.client.0.smithi172.stderr:waiting osd.0 seq 25769803807 2024-05-08T17:14:27.025 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-05-08T17:14:27.440 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 25769803807 -lt 25769803807 2024-05-08T17:14:27.440 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-08T17:14:27.441 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-176093659139 2024-05-08T17:14:27.441 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-08T17:14:27.442 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-05-08T17:14:27.443 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-176093659139 2024-05-08T17:14:27.443 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-08T17:14:27.445 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=176093659139 2024-05-08T17:14:27.445 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 176093659139' 2024-05-08T17:14:27.445 INFO:tasks.workunit.client.0.smithi172.stderr:waiting osd.1 seq 176093659139 2024-05-08T17:14:27.445 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-05-08T17:14:27.847 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 176093659139 -lt 176093659139 2024-05-08T17:14:27.847 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-08T17:14:27.848 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-81604378642 2024-05-08T17:14:27.848 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-08T17:14:27.849 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-05-08T17:14:27.850 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-81604378642 2024-05-08T17:14:27.850 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-08T17:14:27.852 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=81604378642 2024-05-08T17:14:27.852 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 81604378642' 2024-05-08T17:14:27.852 INFO:tasks.workunit.client.0.smithi172.stderr:waiting osd.2 seq 81604378642 2024-05-08T17:14:27.852 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-05-08T17:14:28.248 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 81604378642 -lt 81604378642 2024-05-08T17:14:28.248 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-08T17:14:28.249 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-115964117004 2024-05-08T17:14:28.249 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-08T17:14:28.250 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-05-08T17:14:28.251 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-115964117004 2024-05-08T17:14:28.251 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-08T17:14:28.253 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=115964117004 2024-05-08T17:14:28.253 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 115964117004' 2024-05-08T17:14:28.253 INFO:tasks.workunit.client.0.smithi172.stderr:waiting osd.3 seq 115964117004 2024-05-08T17:14:28.253 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-05-08T17:14:28.657 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 115964117004 -lt 115964117004 2024-05-08T17:14:28.658 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-05-08T17:14:28.658 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-05-08T17:14:28.659 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-05-08T17:14:29.235 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-05-08T17:14:29.235 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-05-08T17:14:29.236 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-05-08T17:14:29.236 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-05-08T17:14:29.236 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-05-08T17:14:29.236 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-05-08T17:14:29.237 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-05-08T17:14:29.237 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-05-08T17:14:29.661 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-05-08T17:14:29.662 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-05-08T17:14:29.663 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-05-08T17:14:29.663 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-05-08T17:14:30.179 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-05-08T17:14:30.179 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-05-08T17:14:30.179 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-05-08T17:14:30.179 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:161: rados_put_get_data: rados_get td/test-erasure-eio pool-jerasure obj-remove-81110-1 2024-05-08T17:14:30.179 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2024-05-08T17:14:30.179 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2024-05-08T17:14:30.180 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=obj-remove-81110-1 2024-05-08T17:14:30.180 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=ok 2024-05-08T17:14:30.180 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' ok = fail ']' 2024-05-08T17:14:30.180 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:120: rados_get: rados --pool pool-jerasure get obj-remove-81110-1 td/test-erasure-eio/COPY 2024-05-08T17:14:30.259 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:121: rados_get: diff td/test-erasure-eio/ORIGINAL td/test-erasure-eio/COPY 2024-05-08T17:14:30.260 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:122: rados_get: rm td/test-erasure-eio/COPY 2024-05-08T17:14:30.261 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:163: rados_put_get_data: '[' '' = recovery ']' 2024-05-08T17:14:30.262 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:181: rados_put_get_data: expr 1 + 1 2024-05-08T17:14:30.264 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:181: rados_put_get_data: shard_id=2 2024-05-08T17:14:30.264 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:182: rados_put_get_data: inject_remove ec data pool-jerasure obj-remove-81110-1 td/test-erasure-eio 2 2024-05-08T17:14:30.264 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:127: inject_remove: local pooltype=ec 2024-05-08T17:14:30.264 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:128: inject_remove: shift 2024-05-08T17:14:30.264 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:129: inject_remove: local which=data 2024-05-08T17:14:30.264 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:130: inject_remove: shift 2024-05-08T17:14:30.265 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:131: inject_remove: local poolname=pool-jerasure 2024-05-08T17:14:30.265 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:132: inject_remove: shift 2024-05-08T17:14:30.265 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:133: inject_remove: local objname=obj-remove-81110-1 2024-05-08T17:14:30.265 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:134: inject_remove: shift 2024-05-08T17:14:30.265 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:135: inject_remove: local dir=td/test-erasure-eio 2024-05-08T17:14:30.265 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:136: inject_remove: shift 2024-05-08T17:14:30.265 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:137: inject_remove: local shard_id=2 2024-05-08T17:14:30.265 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:138: inject_remove: shift 2024-05-08T17:14:30.265 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:140: inject_remove: get_osds pool-jerasure obj-remove-81110-1 2024-05-08T17:14:30.265 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2024-05-08T17:14:30.265 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-remove-81110-1 2024-05-08T17:14:30.266 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-05-08T17:14:30.266 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure obj-remove-81110-1 2024-05-08T17:14:30.706 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-05-08T17:14:30.706 INFO:tasks.workunit.client.0.smithi172.stderr:1 2024-05-08T17:14:30.706 INFO:tasks.workunit.client.0.smithi172.stderr:2' 2024-05-08T17:14:30.706 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 2 2024-05-08T17:14:30.708 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:140: inject_remove: initial_osds=('3' '1' '2') 2024-05-08T17:14:30.708 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:140: inject_remove: local -a initial_osds 2024-05-08T17:14:30.708 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:141: inject_remove: local osd_id=2 2024-05-08T17:14:30.708 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:142: inject_remove: objectstore_tool td/test-erasure-eio 2 obj-remove-81110-1 remove 2024-05-08T17:14:30.709 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/test-erasure-eio 2024-05-08T17:14:30.709 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-05-08T17:14:30.709 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=2 2024-05-08T17:14:30.709 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-05-08T17:14:30.709 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/test-erasure-eio 2 obj-remove-81110-1 remove 2024-05-08T17:14:30.709 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/test-erasure-eio 2024-05-08T17:14:30.709 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-05-08T17:14:30.709 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=2 2024-05-08T17:14:30.709 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-05-08T17:14:30.709 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/test-erasure-eio TERM osd.2 2024-05-08T17:14:30.709 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-08T17:14:30.709 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-08T17:14:30.709 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-08T17:14:30.709 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-08T17:14:30.709 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-08T17:14:31.017 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-08T17:14:31.017 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/test-erasure-eio 2 obj-remove-81110-1 remove 2024-05-08T17:14:31.018 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/test-erasure-eio 2024-05-08T17:14:31.018 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-05-08T17:14:31.018 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=2 2024-05-08T17:14:31.018 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-05-08T17:14:31.018 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/test-erasure-eio/2 2024-05-08T17:14:31.018 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/test-erasure-eio/2 obj-remove-81110-1 remove 2024-05-08T17:14:32.798 INFO:tasks.workunit.client.0.smithi172.stdout:remove 2#2:d6f72261:::obj-remove-81110-1:head# 2024-05-08T17:14:33.335 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/test-erasure-eio 2 2024-05-08T17:14:33.335 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/test-erasure-eio 2024-05-08T17:14:33.335 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-05-08T17:14:33.335 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-05-08T17:14:33.335 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-05-08T17:14:33.335 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/test-erasure-eio/2 2024-05-08T17:14:33.335 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=4457b0f1-028b-482b-8ee3-7ee05861e6cc --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-05-08T17:14:33.335 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-08T17:14:33.335 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-05-08T17:14:33.336 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-08T17:14:33.336 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/2' 2024-05-08T17:14:33.336 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/2/journal' 2024-05-08T17:14:33.336 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-05-08T17:14:33.336 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-05-08T17:14:33.336 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-05-08T17:14:33.336 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-05-08T17:14:33.336 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-08T17:14:33.336 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-08T17:14:33.337 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-08T17:14:33.338 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T17:14:33.338 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81110 2024-05-08T17:14:33.338 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.81110/$cluster-$name.asok' 2024-05-08T17:14:33.338 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.81110/$cluster-$name.asok' 2024-05-08T17:14:33.338 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-05-08T17:14:33.338 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-05-08T17:14:33.338 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-05-08T17:14:33.338 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-08T17:14:33.338 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-08T17:14:33.338 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-08T17:14:33.338 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-08T17:14:33.338 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-05-08T17:14:33.338 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-05-08T17:14:33.338 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/test-erasure-eio/2 2024-05-08T17:14:33.339 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-05-08T17:14:33.339 INFO:tasks.workunit.client.0.smithi172.stderr:start osd.2 2024-05-08T17:14:33.339 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=4457b0f1-028b-482b-8ee3-7ee05861e6cc --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81110/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-05-08T17:14:33.339 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/test-erasure-eio/2/whoami 2024-05-08T17:14:33.340 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-05-08T17:14:33.341 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-05-08T17:14:33.341 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-05-08T17:14:33.341 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-05-08T17:14:33.374 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:14:33.375+0000 7fdf4cd7f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T17:14:33.376 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:14:33.379+0000 7fdf4cd7f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T17:14:33.377 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:14:33.379+0000 7fdf4cd7f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T17:14:33.771 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-05-08T17:14:33.772 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-08T17:14:33.772 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-05-08T17:14:33.772 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-08T17:14:33.772 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-08T17:14:33.772 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:14:33.772 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-08T17:14:33.772 INFO:tasks.workunit.client.0.smithi172.stderr:0 2024-05-08T17:14:33.772 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:14:33.772 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T17:14:33.937 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:14:33.939+0000 7fdf4cd7f7c0 -1 Falling back to public interface 2024-05-08T17:14:34.245 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:14:35.247 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:14:35.247 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:14:35.247 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-08T17:14:35.247 INFO:tasks.workunit.client.0.smithi172.stderr:1 2024-05-08T17:14:35.248 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:14:35.248 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T17:14:35.662 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:14:36.573 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:14:36.575+0000 7fdf4cd7f7c0 -1 osd.2 43 log_to_monitors true 2024-05-08T17:14:36.664 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:14:36.664 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:14:36.664 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-08T17:14:36.664 INFO:tasks.workunit.client.0.smithi172.stderr:2 2024-05-08T17:14:36.665 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:14:36.665 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T17:14:37.125 INFO:tasks.workunit.client.0.smithi172.stderr:osd.2 up in weight 1 up_from 48 up_thru 37 down_at 44 last_clean_interval [19,43) [v2:127.0.0.1:6818/1363364132,v1:127.0.0.1:6819/1363364132] [v2:127.0.0.1:6820/1363364132,v1:127.0.0.1:6821/1363364132] exists,up 5a9950b0-f3fa-42d4-8f6b-9780b2f59359 2024-05-08T17:14:37.126 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-08T17:14:37.126 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-08T17:14:37.126 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-08T17:14:37.126 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-05-08T17:14:37.126 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-05-08T17:14:37.126 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-05-08T17:14:37.126 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-05-08T17:14:37.126 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-05-08T17:14:37.127 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-05-08T17:14:37.127 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-05-08T17:14:37.128 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-05-08T17:14:37.128 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-05-08T17:14:37.128 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-05-08T17:14:37.225 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-05-08T17:14:37.225 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-05-08T17:14:37.226 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-05-08T17:14:37.226 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-05-08T17:14:37.226 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-05-08T17:14:37.226 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-05-08T17:14:37.816 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-05-08T17:14:37.816 INFO:tasks.workunit.client.0.smithi172.stderr:1 2024-05-08T17:14:37.816 INFO:tasks.workunit.client.0.smithi172.stderr:2 2024-05-08T17:14:37.816 INFO:tasks.workunit.client.0.smithi172.stderr:3' 2024-05-08T17:14:37.816 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-05-08T17:14:37.816 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T17:14:37.817 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-05-08T17:14:37.988 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=25769803811 2024-05-08T17:14:37.988 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 25769803811 2024-05-08T17:14:37.988 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803811' 2024-05-08T17:14:37.988 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T17:14:37.988 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-05-08T17:14:38.176 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=176093659143 2024-05-08T17:14:38.177 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 176093659143 2024-05-08T17:14:38.177 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803811 1-176093659143' 2024-05-08T17:14:38.177 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T17:14:38.177 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-05-08T17:14:38.336 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=206158430211 2024-05-08T17:14:38.336 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 206158430211 2024-05-08T17:14:38.336 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803811 1-176093659143 2-206158430211' 2024-05-08T17:14:38.336 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T17:14:38.336 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-05-08T17:14:38.494 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=115964117008 2024-05-08T17:14:38.494 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 115964117008 2024-05-08T17:14:38.494 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803811 1-176093659143 2-206158430211 3-115964117008' 2024-05-08T17:14:38.494 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-08T17:14:38.495 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-08T17:14:38.495 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-25769803811 2024-05-08T17:14:38.495 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-05-08T17:14:38.496 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-25769803811 2024-05-08T17:14:38.496 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-08T17:14:38.497 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=25769803811 2024-05-08T17:14:38.498 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 25769803811' 2024-05-08T17:14:38.498 INFO:tasks.workunit.client.0.smithi172.stderr:waiting osd.0 seq 25769803811 2024-05-08T17:14:38.498 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-05-08T17:14:38.944 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 25769803809 -lt 25769803811 2024-05-08T17:14:38.944 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-05-08T17:14:39.945 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-05-08T17:14:39.946 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-05-08T17:14:40.362 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 25769803812 -lt 25769803811 2024-05-08T17:14:40.362 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-08T17:14:40.364 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-176093659143 2024-05-08T17:14:40.364 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-08T17:14:40.365 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-05-08T17:14:40.366 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-176093659143 2024-05-08T17:14:40.366 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-08T17:14:40.367 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=176093659143 2024-05-08T17:14:40.367 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 176093659143' 2024-05-08T17:14:40.367 INFO:tasks.workunit.client.0.smithi172.stderr:waiting osd.1 seq 176093659143 2024-05-08T17:14:40.367 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-05-08T17:14:40.774 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 176093659143 -lt 176093659143 2024-05-08T17:14:40.774 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-08T17:14:40.775 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-206158430211 2024-05-08T17:14:40.775 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-08T17:14:40.776 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-05-08T17:14:40.777 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-206158430211 2024-05-08T17:14:40.777 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-08T17:14:40.778 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=206158430211 2024-05-08T17:14:40.778 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 206158430211' 2024-05-08T17:14:40.778 INFO:tasks.workunit.client.0.smithi172.stderr:waiting osd.2 seq 206158430211 2024-05-08T17:14:40.779 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-05-08T17:14:41.224 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 206158430211 -lt 206158430211 2024-05-08T17:14:41.224 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-08T17:14:41.225 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-115964117008 2024-05-08T17:14:41.225 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-08T17:14:41.227 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-05-08T17:14:41.228 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-115964117008 2024-05-08T17:14:41.228 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-08T17:14:41.230 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=115964117008 2024-05-08T17:14:41.230 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 115964117008' 2024-05-08T17:14:41.230 INFO:tasks.workunit.client.0.smithi172.stderr:waiting osd.3 seq 115964117008 2024-05-08T17:14:41.230 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-05-08T17:14:41.647 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 115964117009 -lt 115964117008 2024-05-08T17:14:41.648 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-05-08T17:14:41.648 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-05-08T17:14:41.649 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-05-08T17:14:42.176 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-05-08T17:14:42.176 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-05-08T17:14:42.177 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-05-08T17:14:42.177 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-05-08T17:14:42.177 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-05-08T17:14:42.177 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-05-08T17:14:42.177 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-05-08T17:14:42.177 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-05-08T17:14:42.567 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-05-08T17:14:42.567 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-05-08T17:14:42.568 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-05-08T17:14:42.568 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-05-08T17:14:43.109 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-05-08T17:14:43.109 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-05-08T17:14:43.109 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-05-08T17:14:43.109 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:183: rados_put_get_data: rados_get td/test-erasure-eio pool-jerasure obj-remove-81110-1 fail 2024-05-08T17:14:43.109 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2024-05-08T17:14:43.110 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2024-05-08T17:14:43.110 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=obj-remove-81110-1 2024-05-08T17:14:43.110 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=fail 2024-05-08T17:14:43.110 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' fail = fail ']' 2024-05-08T17:14:43.110 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:114: rados_get: rados --pool pool-jerasure get obj-remove-81110-1 td/test-erasure-eio/COPY 2024-05-08T17:14:43.147 INFO:tasks.workunit.client.0.smithi172.stderr:error getting pool-jerasure/obj-remove-81110-1: (5) Input/output error 2024-05-08T17:14:43.149 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:115: rados_get: return 2024-05-08T17:14:43.149 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:184: rados_put_get_data: rm td/test-erasure-eio/ORIGINAL 2024-05-08T17:14:43.150 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:290: TEST_rados_get_subread_missing: delete_erasure_coded_pool pool-jerasure 2024-05-08T17:14:43.150 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:84: delete_erasure_coded_pool: local poolname=pool-jerasure 2024-05-08T17:14:43.150 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:85: delete_erasure_coded_pool: ceph osd pool delete pool-jerasure pool-jerasure --yes-i-really-really-mean-it 2024-05-08T17:14:43.623 INFO:tasks.workunit.client.0.smithi172.stderr:pool 'pool-jerasure' does not exist 2024-05-08T17:14:43.639 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:86: delete_erasure_coded_pool: ceph osd erasure-code-profile rm myprofile 2024-05-08T17:14:44.188 INFO:tasks.workunit.client.0.smithi172.stderr:erasure-code-profile myprofile does not exist 2024-05-08T17:14:44.204 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:42: run: teardown td/test-erasure-eio 2024-05-08T17:14:44.204 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-eio 2024-05-08T17:14:44.204 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-05-08T17:14:44.204 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-eio KILL 2024-05-08T17:14:44.205 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-08T17:14:44.205 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-08T17:14:44.205 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-08T17:14:44.205 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-08T17:14:44.205 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-08T17:14:44.516 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-08T17:14:44.517 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-05-08T17:14:44.518 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-05-08T17:14:44.519 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-05-08T17:14:44.520 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-05-08T17:14:44.520 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-05-08T17:14:44.521 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-05-08T17:14:44.521 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-08T17:14:44.521 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-05-08T17:14:44.522 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-05-08T17:14:44.522 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-08T17:14:44.523 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-05-08T17:14:44.525 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-05-08T17:14:44.525 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-eio 2024-05-08T17:14:44.551 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-05-08T17:14:44.551 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T17:14:44.551 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81110 2024-05-08T17:14:44.552 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.81110 2024-05-08T17:14:44.553 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-05-08T17:14:44.554 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-05-08T17:14:44.554 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:32: run: for func in $funcs 2024-05-08T17:14:44.554 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:33: run: setup td/test-erasure-eio 2024-05-08T17:14:44.554 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/test-erasure-eio 2024-05-08T17:14:44.554 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/test-erasure-eio 2024-05-08T17:14:44.554 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-eio 2024-05-08T17:14:44.554 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-05-08T17:14:44.554 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-eio KILL 2024-05-08T17:14:44.554 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-08T17:14:44.554 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-08T17:14:44.554 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-08T17:14:44.554 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-08T17:14:44.554 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-08T17:14:44.556 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-08T17:14:44.557 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-05-08T17:14:44.557 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-05-08T17:14:44.558 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-05-08T17:14:44.559 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-05-08T17:14:44.559 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-05-08T17:14:44.560 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-05-08T17:14:44.561 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-08T17:14:44.561 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-05-08T17:14:44.561 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-05-08T17:14:44.561 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-08T17:14:44.562 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-05-08T17:14:44.564 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-05-08T17:14:44.564 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-eio 2024-05-08T17:14:44.565 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-05-08T17:14:44.565 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T17:14:44.565 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81110 2024-05-08T17:14:44.566 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.81110 2024-05-08T17:14:44.566 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-05-08T17:14:44.566 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-05-08T17:14:44.566 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/test-erasure-eio 2024-05-08T17:14:44.568 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-05-08T17:14:44.568 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T17:14:44.568 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81110 2024-05-08T17:14:44.568 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.81110 2024-05-08T17:14:44.571 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-05-08T17:14:44.571 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-05-08T17:14:44.571 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-05-08T17:14:44.571 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/test-erasure-eio 1' TERM HUP INT 2024-05-08T17:14:44.571 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:34: run: run_mon td/test-erasure-eio a 2024-05-08T17:14:44.571 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/test-erasure-eio 2024-05-08T17:14:44.571 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-05-08T17:14:44.571 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-05-08T17:14:44.571 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-05-08T17:14:44.571 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/test-erasure-eio/a 2024-05-08T17:14:44.571 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-eio/a --run-dir=td/test-erasure-eio 2024-05-08T17:14:44.965 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-05-08T17:14:44.965 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-08T17:14:44.965 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-08T17:14:44.965 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-08T17:14:44.965 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T17:14:44.965 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81110 2024-05-08T17:14:44.965 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.81110/$cluster-$name.asok' 2024-05-08T17:14:44.966 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-eio/a '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.81110/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-eio/log --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-05-08T17:14:45.176 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-05-08T17:14:45.177 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-05-08T17:14:45.177 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-05-08T17:14:45.177 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-05-08T17:14:45.177 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-05-08T17:14:45.178 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-05-08T17:14:45.178 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-05-08T17:14:45.178 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-05-08T17:14:45.178 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-05-08T17:14:45.179 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-08T17:14:45.179 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T17:14:45.179 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81110 2024-05-08T17:14:45.179 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.81110/ceph-mon.a.asok 2024-05-08T17:14:45.179 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-05-08T17:14:45.180 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.81110/ceph-mon.a.asok config get fsid 2024-05-08T17:14:45.296 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-05-08T17:14:45.296 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-05-08T17:14:45.296 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-05-08T17:14:45.296 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-05-08T17:14:45.297 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-05-08T17:14:45.298 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-05-08T17:14:45.298 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-05-08T17:14:45.298 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-05-08T17:14:45.298 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-08T17:14:45.298 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T17:14:45.298 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81110 2024-05-08T17:14:45.299 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.81110/ceph-mon.a.asok 2024-05-08T17:14:45.299 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-05-08T17:14:45.299 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.81110/ceph-mon.a.asok config get mon_host 2024-05-08T17:14:45.401 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:35: run: run_mgr td/test-erasure-eio x 2024-05-08T17:14:45.401 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/test-erasure-eio 2024-05-08T17:14:45.401 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-05-08T17:14:45.402 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-05-08T17:14:45.402 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-05-08T17:14:45.402 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/test-erasure-eio/x 2024-05-08T17:14:45.402 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-05-08T17:14:45.895 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-05-08T17:14:45.895 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-08T17:14:45.895 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-08T17:14:45.895 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-08T17:14:45.895 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T17:14:45.895 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81110 2024-05-08T17:14:45.895 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.81110/$cluster-$name.asok' 2024-05-08T17:14:45.896 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-05-08T17:14:45.897 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-eio/x '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.81110/$cluster-$name.asok' --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-05-08T17:14:45.927 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:36: run: create_pool rbd 4 2024-05-08T17:14:45.927 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-05-08T17:14:46.220 INFO:tasks.workunit.client.0.smithi172.stderr:pool 'rbd' already exists 2024-05-08T17:14:46.235 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-05-08T17:14:47.239 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: get_asok_path mon.a 2024-05-08T17:14:47.239 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-05-08T17:14:47.239 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-05-08T17:14:47.239 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-08T17:14:47.240 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T17:14:47.240 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81110 2024-05-08T17:14:47.240 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.81110/ceph-mon.a.asok 2024-05-08T17:14:47.240 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: CEPH_ARGS= 2024-05-08T17:14:47.240 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: ceph --admin-daemon /tmp/ceph-asok.81110/ceph-mon.a.asok log flush 2024-05-08T17:14:47.338 INFO:tasks.workunit.client.0.smithi172.stdout:{} 2024-05-08T17:14:47.351 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:40: run: grep 'load: jerasure.*lrc' td/test-erasure-eio/mon.a.log 2024-05-08T17:14:47.353 INFO:tasks.workunit.client.0.smithi172.stdout:2024-05-08T17:14:45.011+0000 7fb3d17eac80 0 load: jerasure load: lrc 2024-05-08T17:14:47.354 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:41: run: TEST_rados_get_with_subreadall_eio_shard_0 td/test-erasure-eio 2024-05-08T17:14:47.354 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:330: TEST_rados_get_with_subreadall_eio_shard_0: local dir=td/test-erasure-eio 2024-05-08T17:14:47.354 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:331: TEST_rados_get_with_subreadall_eio_shard_0: local shard_id=0 2024-05-08T17:14:47.354 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:333: TEST_rados_get_with_subreadall_eio_shard_0: setup_osds 4 2024-05-08T17:14:47.354 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:47: setup_osds: local count=4 2024-05-08T17:14:47.354 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:48: setup_osds: shift 2024-05-08T17:14:47.355 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: expr 4 - 1 2024-05-08T17:14:47.356 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: seq 0 3 2024-05-08T17:14:47.359 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-05-08T17:14:47.359 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 0 2024-05-08T17:14:47.359 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-05-08T17:14:47.359 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-05-08T17:14:47.359 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-05-08T17:14:47.359 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-05-08T17:14:47.359 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/0 2024-05-08T17:14:47.359 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=4457b0f1-028b-482b-8ee3-7ee05861e6cc --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-05-08T17:14:47.359 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-08T17:14:47.359 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-05-08T17:14:47.359 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-08T17:14:47.359 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/0' 2024-05-08T17:14:47.359 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/0/journal' 2024-05-08T17:14:47.360 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-05-08T17:14:47.360 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-05-08T17:14:47.360 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-05-08T17:14:47.360 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-05-08T17:14:47.360 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-08T17:14:47.360 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-08T17:14:47.360 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-08T17:14:47.360 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T17:14:47.360 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81110 2024-05-08T17:14:47.360 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.81110/$cluster-$name.asok' 2024-05-08T17:14:47.362 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.81110/$cluster-$name.asok' 2024-05-08T17:14:47.362 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-05-08T17:14:47.362 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-05-08T17:14:47.362 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-05-08T17:14:47.362 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-05-08T17:14:47.362 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-05-08T17:14:47.362 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-08T17:14:47.362 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-08T17:14:47.362 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-08T17:14:47.362 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-08T17:14:47.362 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-05-08T17:14:47.362 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-05-08T17:14:47.363 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/0 2024-05-08T17:14:47.363 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-05-08T17:14:47.363 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=0ba02da4-6a81-4fc3-8710-2ce80a2bb7ae 2024-05-08T17:14:47.363 INFO:tasks.workunit.client.0.smithi172.stdout:add osd0 0ba02da4-6a81-4fc3-8710-2ce80a2bb7ae 2024-05-08T17:14:47.363 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 0ba02da4-6a81-4fc3-8710-2ce80a2bb7ae' 2024-05-08T17:14:47.363 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-05-08T17:14:47.379 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAHsztmcNX9FhAALKnAP3rT+GYJCrSRRZaFWA== 2024-05-08T17:14:47.379 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAHsztmcNX9FhAALKnAP3rT+GYJCrSRRZaFWA=="}' 2024-05-08T17:14:47.380 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 0ba02da4-6a81-4fc3-8710-2ce80a2bb7ae -i td/test-erasure-eio/0/new.json 2024-05-08T17:14:47.664 INFO:tasks.workunit.client.0.smithi172.stdout:0 2024-05-08T17:14:47.679 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/0/new.json 2024-05-08T17:14:47.680 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=4457b0f1-028b-482b-8ee3-7ee05861e6cc --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81110/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAHsztmcNX9FhAALKnAP3rT+GYJCrSRRZaFWA== --osd-uuid 0ba02da4-6a81-4fc3-8710-2ce80a2bb7ae 2024-05-08T17:14:47.708 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:14:47.711+0000 7ff23d8d27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T17:14:47.732 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:14:47.739+0000 7ff23d8d27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T17:14:47.733 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:14:47.739+0000 7ff23d8d27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T17:14:47.734 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:14:47.739+0000 7ff23d8d27c0 -1 bluestore(td/test-erasure-eio/0/block) _read_bdev_label failed to open td/test-erasure-eio/0/block: (2) No such file or directory 2024-05-08T17:14:47.734 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:14:47.739+0000 7ff23d8d27c0 -1 bluestore(td/test-erasure-eio/0/block) _read_bdev_label failed to open td/test-erasure-eio/0/block: (2) No such file or directory 2024-05-08T17:14:47.734 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:14:47.739+0000 7ff23d8d27c0 -1 bluestore(td/test-erasure-eio/0/block) _read_bdev_label failed to open td/test-erasure-eio/0/block: (2) No such file or directory 2024-05-08T17:14:47.772 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:14:47.779+0000 7ff23d8d27c0 -1 bluestore(td/test-erasure-eio/0) _read_fsid unparsable uuid 2024-05-08T17:14:53.449 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/0/keyring 2024-05-08T17:14:53.450 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-05-08T17:14:53.451 INFO:tasks.workunit.client.0.smithi172.stdout:adding osd0 key to auth repository 2024-05-08T17:14:53.451 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-05-08T17:14:53.451 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-05-08T17:14:53.994 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-05-08T17:14:53.994 INFO:tasks.workunit.client.0.smithi172.stdout:start osd.0 2024-05-08T17:14:53.995 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=4457b0f1-028b-482b-8ee3-7ee05861e6cc --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81110/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-05-08T17:14:53.995 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-05-08T17:14:53.995 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-05-08T17:14:53.995 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-05-08T17:14:54.026 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:14:54.031+0000 7efdd79207c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T17:14:54.045 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:14:54.051+0000 7efdd79207c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T17:14:54.047 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:14:54.051+0000 7efdd79207c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T17:14:54.397 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-05-08T17:14:54.397 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-08T17:14:54.397 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-05-08T17:14:54.397 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-08T17:14:54.397 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-08T17:14:54.397 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:14:54.398 INFO:tasks.workunit.client.0.smithi172.stdout:0 2024-05-08T17:14:54.398 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-08T17:14:54.398 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:14:54.398 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T17:14:54.610 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:14:54.615+0000 7efdd79207c0 -1 Falling back to public interface 2024-05-08T17:14:54.789 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:14:55.791 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:14:55.791 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:14:55.791 INFO:tasks.workunit.client.0.smithi172.stdout:1 2024-05-08T17:14:55.792 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-08T17:14:55.792 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:14:55.792 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T17:14:56.195 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:14:57.062 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:14:57.067+0000 7efdd79207c0 -1 osd.0 0 log_to_monitors true 2024-05-08T17:14:57.196 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:14:57.197 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:14:57.197 INFO:tasks.workunit.client.0.smithi172.stdout:2 2024-05-08T17:14:57.197 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-08T17:14:57.197 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:14:57.198 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T17:14:57.694 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:14:58.696 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:14:58.696 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:14:58.696 INFO:tasks.workunit.client.0.smithi172.stdout:3 2024-05-08T17:14:58.697 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-05-08T17:14:58.697 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:14:58.697 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T17:14:59.169 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:15:00.171 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:15:00.171 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:15:00.171 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-05-08T17:15:00.171 INFO:tasks.workunit.client.0.smithi172.stdout:4 2024-05-08T17:15:00.172 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:15:00.172 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T17:15:00.570 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:15:01.572 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:15:01.572 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:15:01.572 INFO:tasks.workunit.client.0.smithi172.stdout:5 2024-05-08T17:15:01.573 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-05-08T17:15:01.573 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:15:01.573 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T17:15:02.055 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:15:03.056 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:15:03.056 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:15:03.057 INFO:tasks.workunit.client.0.smithi172.stdout:6 2024-05-08T17:15:03.057 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-05-08T17:15:03.057 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:15:03.057 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T17:15:03.468 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:15:04.469 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:15:04.469 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:15:04.469 INFO:tasks.workunit.client.0.smithi172.stdout:7 2024-05-08T17:15:04.470 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-05-08T17:15:04.470 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:15:04.470 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T17:15:04.953 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:15:05.955 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:15:05.955 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:15:05.955 INFO:tasks.workunit.client.0.smithi172.stdout:8 2024-05-08T17:15:05.956 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-05-08T17:15:05.956 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:15:05.956 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T17:15:06.486 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:15:07.488 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:15:07.488 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:15:07.488 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-05-08T17:15:07.488 INFO:tasks.workunit.client.0.smithi172.stdout:9 2024-05-08T17:15:07.489 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:15:07.489 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T17:15:07.946 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:15:08.948 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:15:08.948 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:15:08.948 INFO:tasks.workunit.client.0.smithi172.stdout:10 2024-05-08T17:15:08.949 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-05-08T17:15:08.949 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:15:08.949 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T17:15:09.682 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:15:10.684 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:15:10.684 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:15:10.684 INFO:tasks.workunit.client.0.smithi172.stdout:11 2024-05-08T17:15:10.685 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-05-08T17:15:10.685 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:15:10.685 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T17:15:11.526 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:15:12.527 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:15:12.527 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:15:12.528 INFO:tasks.workunit.client.0.smithi172.stdout:12 2024-05-08T17:15:12.528 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-05-08T17:15:12.528 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:15:12.528 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T17:15:13.615 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:15:14.617 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:15:14.617 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:15:14.617 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-05-08T17:15:14.617 INFO:tasks.workunit.client.0.smithi172.stdout:13 2024-05-08T17:15:14.617 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:15:14.618 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T17:15:15.637 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:15:16.639 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:15:16.639 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:15:16.639 INFO:tasks.workunit.client.0.smithi172.stdout:14 2024-05-08T17:15:16.640 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-05-08T17:15:16.640 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:15:16.640 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T17:15:17.831 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:15:18.833 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:15:18.833 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:15:18.833 INFO:tasks.workunit.client.0.smithi172.stdout:15 2024-05-08T17:15:18.834 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-05-08T17:15:18.834 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:15:18.834 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T17:15:19.442 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:15:20.444 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:15:20.444 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:15:20.445 INFO:tasks.workunit.client.0.smithi172.stdout:16 2024-05-08T17:15:20.445 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-05-08T17:15:20.445 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:15:20.445 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T17:15:20.852 INFO:tasks.workunit.client.0.smithi172.stdout:osd.0 up in weight 1 up_from 6 up_thru 6 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2625731450,v1:127.0.0.1:6803/2625731450] [v2:127.0.0.1:6804/2625731450,v1:127.0.0.1:6805/2625731450] exists,up 0ba02da4-6a81-4fc3-8710-2ce80a2bb7ae 2024-05-08T17:15:20.853 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-08T17:15:20.853 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-08T17:15:20.853 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-08T17:15:20.854 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-05-08T17:15:20.854 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 1 2024-05-08T17:15:20.854 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-05-08T17:15:20.854 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-05-08T17:15:20.854 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-05-08T17:15:20.854 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-05-08T17:15:20.854 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/1 2024-05-08T17:15:20.854 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=4457b0f1-028b-482b-8ee3-7ee05861e6cc --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-05-08T17:15:20.855 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-08T17:15:20.855 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-05-08T17:15:20.855 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-08T17:15:20.855 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/1' 2024-05-08T17:15:20.855 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/1/journal' 2024-05-08T17:15:20.855 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-05-08T17:15:20.855 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-05-08T17:15:20.855 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-05-08T17:15:20.855 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-05-08T17:15:20.855 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-08T17:15:20.855 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-08T17:15:20.857 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-08T17:15:20.857 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T17:15:20.857 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81110 2024-05-08T17:15:20.858 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.81110/$cluster-$name.asok' 2024-05-08T17:15:20.858 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.81110/$cluster-$name.asok' 2024-05-08T17:15:20.858 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-05-08T17:15:20.858 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-05-08T17:15:20.858 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-05-08T17:15:20.858 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-05-08T17:15:20.858 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-05-08T17:15:20.858 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-08T17:15:20.858 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-08T17:15:20.858 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-08T17:15:20.858 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-08T17:15:20.858 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-05-08T17:15:20.858 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-05-08T17:15:20.858 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/1 2024-05-08T17:15:20.859 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-05-08T17:15:20.859 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=1259b60b-a74d-4554-91ba-f7cbab20a4dc 2024-05-08T17:15:20.859 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 1259b60b-a74d-4554-91ba-f7cbab20a4dc' 2024-05-08T17:15:20.859 INFO:tasks.workunit.client.0.smithi172.stdout:add osd1 1259b60b-a74d-4554-91ba-f7cbab20a4dc 2024-05-08T17:15:20.859 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-05-08T17:15:20.880 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAosztmaU/RNBAA9hTz7fov2D6ke3oWna6qjA== 2024-05-08T17:15:20.880 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAosztmaU/RNBAA9hTz7fov2D6ke3oWna6qjA=="}' 2024-05-08T17:15:20.880 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 1259b60b-a74d-4554-91ba-f7cbab20a4dc -i td/test-erasure-eio/1/new.json 2024-05-08T17:15:21.386 INFO:tasks.workunit.client.0.smithi172.stdout:1 2024-05-08T17:15:21.406 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/1/new.json 2024-05-08T17:15:21.406 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=4457b0f1-028b-482b-8ee3-7ee05861e6cc --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81110/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAosztmaU/RNBAA9hTz7fov2D6ke3oWna6qjA== --osd-uuid 1259b60b-a74d-4554-91ba-f7cbab20a4dc 2024-05-08T17:15:21.433 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:15:21.438+0000 7fe1cdb287c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T17:15:21.435 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:15:21.438+0000 7fe1cdb287c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T17:15:21.436 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:15:21.442+0000 7fe1cdb287c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T17:15:21.437 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:15:21.442+0000 7fe1cdb287c0 -1 bluestore(td/test-erasure-eio/1/block) _read_bdev_label failed to open td/test-erasure-eio/1/block: (2) No such file or directory 2024-05-08T17:15:21.437 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:15:21.442+0000 7fe1cdb287c0 -1 bluestore(td/test-erasure-eio/1/block) _read_bdev_label failed to open td/test-erasure-eio/1/block: (2) No such file or directory 2024-05-08T17:15:21.437 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:15:21.442+0000 7fe1cdb287c0 -1 bluestore(td/test-erasure-eio/1/block) _read_bdev_label failed to open td/test-erasure-eio/1/block: (2) No such file or directory 2024-05-08T17:15:21.508 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:15:21.514+0000 7fe1cdb287c0 -1 bluestore(td/test-erasure-eio/1) _read_fsid unparsable uuid 2024-05-08T17:15:27.636 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/1/keyring 2024-05-08T17:15:27.636 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-05-08T17:15:27.637 INFO:tasks.workunit.client.0.smithi172.stdout:adding osd1 key to auth repository 2024-05-08T17:15:27.637 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-05-08T17:15:27.637 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-05-08T17:15:28.190 INFO:tasks.workunit.client.0.smithi172.stdout:start osd.1 2024-05-08T17:15:28.191 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-05-08T17:15:28.191 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-05-08T17:15:28.191 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-05-08T17:15:28.191 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=4457b0f1-028b-482b-8ee3-7ee05861e6cc --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81110/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-05-08T17:15:28.191 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-05-08T17:15:28.222 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:15:28.226+0000 7fe9649e57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T17:15:28.248 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:15:28.254+0000 7fe9649e57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T17:15:28.249 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:15:28.254+0000 7fe9649e57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T17:15:28.687 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-05-08T17:15:28.687 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-08T17:15:28.687 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-05-08T17:15:28.687 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-08T17:15:28.687 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-08T17:15:28.687 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:15:28.687 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-08T17:15:28.688 INFO:tasks.workunit.client.0.smithi172.stdout:0 2024-05-08T17:15:28.688 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T17:15:28.688 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:15:28.817 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:15:28.822+0000 7fe9649e57c0 -1 Falling back to public interface 2024-05-08T17:15:29.094 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:15:30.096 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:15:30.096 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:15:30.097 INFO:tasks.workunit.client.0.smithi172.stdout:1 2024-05-08T17:15:30.098 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-08T17:15:30.098 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:15:30.098 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T17:15:30.520 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:15:31.472 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:15:31.478+0000 7fe9649e57c0 -1 osd.1 0 log_to_monitors true 2024-05-08T17:15:31.521 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:15:31.521 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:15:31.522 INFO:tasks.workunit.client.0.smithi172.stdout:2 2024-05-08T17:15:31.523 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-08T17:15:31.523 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:15:31.523 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T17:15:32.132 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:15:33.134 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:15:33.134 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:15:33.134 INFO:tasks.workunit.client.0.smithi172.stdout:3 2024-05-08T17:15:33.135 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-05-08T17:15:33.135 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:15:33.135 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T17:15:33.551 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:15:34.552 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:15:34.553 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:15:34.553 INFO:tasks.workunit.client.0.smithi172.stdout:4 2024-05-08T17:15:34.553 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-05-08T17:15:34.553 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:15:34.554 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T17:15:34.961 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:15:35.963 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:15:35.963 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:15:35.963 INFO:tasks.workunit.client.0.smithi172.stdout:5 2024-05-08T17:15:35.964 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-05-08T17:15:35.964 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:15:35.964 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T17:15:36.517 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:15:37.519 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:15:37.519 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:15:37.519 INFO:tasks.workunit.client.0.smithi172.stdout:6 2024-05-08T17:15:37.520 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-05-08T17:15:37.520 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:15:37.520 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T17:15:37.965 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:15:38.967 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:15:38.967 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:15:38.968 INFO:tasks.workunit.client.0.smithi172.stdout:7 2024-05-08T17:15:38.968 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-05-08T17:15:38.968 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:15:38.969 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T17:15:39.394 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:15:40.395 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:15:40.396 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:15:40.396 INFO:tasks.workunit.client.0.smithi172.stdout:8 2024-05-08T17:15:40.396 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-05-08T17:15:40.396 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:15:40.396 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T17:15:40.886 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:15:41.890 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:15:41.890 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:15:41.890 INFO:tasks.workunit.client.0.smithi172.stdout:9 2024-05-08T17:15:41.891 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-05-08T17:15:41.891 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:15:41.891 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T17:15:42.369 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:15:43.372 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:15:43.372 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:15:43.373 INFO:tasks.workunit.client.0.smithi172.stdout:10 2024-05-08T17:15:43.373 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-05-08T17:15:43.373 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:15:43.373 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T17:15:44.376 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:15:45.378 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:15:45.379 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:15:45.379 INFO:tasks.workunit.client.0.smithi172.stdout:11 2024-05-08T17:15:45.379 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-05-08T17:15:45.379 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:15:45.379 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T17:15:46.369 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:15:47.371 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:15:47.371 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:15:47.371 INFO:tasks.workunit.client.0.smithi172.stdout:12 2024-05-08T17:15:47.371 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-05-08T17:15:47.372 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:15:47.372 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T17:15:48.031 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:15:49.033 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:15:49.034 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:15:49.034 INFO:tasks.workunit.client.0.smithi172.stdout:13 2024-05-08T17:15:49.034 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-05-08T17:15:49.034 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:15:49.035 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T17:15:49.880 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:15:50.882 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:15:50.882 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:15:50.883 INFO:tasks.workunit.client.0.smithi172.stdout:14 2024-05-08T17:15:50.883 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-05-08T17:15:50.883 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:15:50.883 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T17:15:52.167 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:15:53.169 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:15:53.169 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:15:53.169 INFO:tasks.workunit.client.0.smithi172.stdout:15 2024-05-08T17:15:53.169 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-05-08T17:15:53.170 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:15:53.170 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T17:15:53.639 INFO:tasks.workunit.client.0.smithi172.stdout:osd.1 up in weight 1 up_from 13 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/3470107868,v1:127.0.0.1:6811/3470107868] [v2:127.0.0.1:6812/3470107868,v1:127.0.0.1:6813/3470107868] exists,up 1259b60b-a74d-4554-91ba-f7cbab20a4dc 2024-05-08T17:15:53.641 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-08T17:15:53.641 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-08T17:15:53.641 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-08T17:15:53.641 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-05-08T17:15:53.641 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 2 2024-05-08T17:15:53.641 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-05-08T17:15:53.641 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-05-08T17:15:53.641 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-05-08T17:15:53.641 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-05-08T17:15:53.641 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/2 2024-05-08T17:15:53.641 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=4457b0f1-028b-482b-8ee3-7ee05861e6cc --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-05-08T17:15:53.641 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-08T17:15:53.641 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-05-08T17:15:53.642 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-08T17:15:53.642 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/2' 2024-05-08T17:15:53.642 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/2/journal' 2024-05-08T17:15:53.642 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-05-08T17:15:53.642 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-05-08T17:15:53.642 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-05-08T17:15:53.642 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-05-08T17:15:53.642 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-08T17:15:53.642 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-08T17:15:53.644 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-08T17:15:53.644 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T17:15:53.644 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81110 2024-05-08T17:15:53.644 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.81110/$cluster-$name.asok' 2024-05-08T17:15:53.644 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.81110/$cluster-$name.asok' 2024-05-08T17:15:53.644 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-05-08T17:15:53.644 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-05-08T17:15:53.644 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-05-08T17:15:53.644 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-05-08T17:15:53.644 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-05-08T17:15:53.645 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-08T17:15:53.645 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-08T17:15:53.645 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-08T17:15:53.645 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-08T17:15:53.645 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-05-08T17:15:53.645 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-05-08T17:15:53.645 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/2 2024-05-08T17:15:53.645 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-05-08T17:15:53.646 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=383cfd0c-3d6e-43a7-8b7c-bcd89e0b2c03 2024-05-08T17:15:53.646 INFO:tasks.workunit.client.0.smithi172.stdout:add osd2 383cfd0c-3d6e-43a7-8b7c-bcd89e0b2c03 2024-05-08T17:15:53.646 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 383cfd0c-3d6e-43a7-8b7c-bcd89e0b2c03' 2024-05-08T17:15:53.646 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-05-08T17:15:53.669 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBJsztmJsM/KBAAVYxHehvOSK6IGxWzY4RdaQ== 2024-05-08T17:15:53.669 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBJsztmJsM/KBAAVYxHehvOSK6IGxWzY4RdaQ=="}' 2024-05-08T17:15:53.669 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 383cfd0c-3d6e-43a7-8b7c-bcd89e0b2c03 -i td/test-erasure-eio/2/new.json 2024-05-08T17:15:54.472 INFO:tasks.workunit.client.0.smithi172.stdout:2 2024-05-08T17:15:54.489 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/2/new.json 2024-05-08T17:15:54.490 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=4457b0f1-028b-482b-8ee3-7ee05861e6cc --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81110/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBJsztmJsM/KBAAVYxHehvOSK6IGxWzY4RdaQ== --osd-uuid 383cfd0c-3d6e-43a7-8b7c-bcd89e0b2c03 2024-05-08T17:15:54.524 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:15:54.529+0000 7fe695d887c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T17:15:54.577 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:15:54.581+0000 7fe695d887c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T17:15:54.580 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:15:54.581+0000 7fe695d887c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T17:15:54.580 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:15:54.581+0000 7fe695d887c0 -1 bluestore(td/test-erasure-eio/2/block) _read_bdev_label failed to open td/test-erasure-eio/2/block: (2) No such file or directory 2024-05-08T17:15:54.580 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:15:54.581+0000 7fe695d887c0 -1 bluestore(td/test-erasure-eio/2/block) _read_bdev_label failed to open td/test-erasure-eio/2/block: (2) No such file or directory 2024-05-08T17:15:54.580 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:15:54.581+0000 7fe695d887c0 -1 bluestore(td/test-erasure-eio/2/block) _read_bdev_label failed to open td/test-erasure-eio/2/block: (2) No such file or directory 2024-05-08T17:15:54.670 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:15:54.673+0000 7fe695d887c0 -1 bluestore(td/test-erasure-eio/2) _read_fsid unparsable uuid 2024-05-08T17:16:00.717 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/2/keyring 2024-05-08T17:16:00.718 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-05-08T17:16:00.722 INFO:tasks.workunit.client.0.smithi172.stdout:adding osd2 key to auth repository 2024-05-08T17:16:00.722 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-05-08T17:16:00.722 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-05-08T17:16:01.281 INFO:tasks.workunit.client.0.smithi172.stdout:start osd.2 2024-05-08T17:16:01.281 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-05-08T17:16:01.281 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=4457b0f1-028b-482b-8ee3-7ee05861e6cc --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81110/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-05-08T17:16:01.282 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-05-08T17:16:01.282 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-05-08T17:16:01.282 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-05-08T17:16:01.312 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:16:01.317+0000 7fb2349247c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T17:16:01.314 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:16:01.317+0000 7fb2349247c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T17:16:01.316 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:16:01.321+0000 7fb2349247c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T17:16:01.743 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-05-08T17:16:01.743 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-08T17:16:01.743 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-05-08T17:16:01.743 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-08T17:16:01.743 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-08T17:16:01.743 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:16:01.743 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-08T17:16:01.743 INFO:tasks.workunit.client.0.smithi172.stdout:0 2024-05-08T17:16:01.744 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:16:01.744 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T17:16:01.885 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:16:01.889+0000 7fb2349247c0 -1 Falling back to public interface 2024-05-08T17:16:02.183 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:16:03.185 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:16:03.186 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:16:03.186 INFO:tasks.workunit.client.0.smithi172.stdout:1 2024-05-08T17:16:03.186 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-08T17:16:03.186 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:16:03.186 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T17:16:03.580 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:16:04.462 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:16:04.465+0000 7fb2349247c0 -1 osd.2 0 log_to_monitors true 2024-05-08T17:16:04.582 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:16:04.582 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:16:04.583 INFO:tasks.workunit.client.0.smithi172.stdout:2 2024-05-08T17:16:04.583 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-08T17:16:04.583 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:16:04.583 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T17:16:05.064 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:16:06.065 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:16:06.065 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:16:06.066 INFO:tasks.workunit.client.0.smithi172.stdout:3 2024-05-08T17:16:06.066 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-05-08T17:16:06.066 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:16:06.066 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T17:16:06.515 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:16:07.517 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:16:07.517 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:16:07.517 INFO:tasks.workunit.client.0.smithi172.stdout:4 2024-05-08T17:16:07.518 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-05-08T17:16:07.518 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:16:07.518 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T17:16:07.960 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:16:08.962 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:16:08.962 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:16:08.962 INFO:tasks.workunit.client.0.smithi172.stdout:5 2024-05-08T17:16:08.963 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-05-08T17:16:08.963 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:16:08.963 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T17:16:09.496 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:16:10.498 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:16:10.498 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:16:10.498 INFO:tasks.workunit.client.0.smithi172.stdout:6 2024-05-08T17:16:10.499 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-05-08T17:16:10.499 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:16:10.499 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T17:16:10.931 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:16:11.933 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:16:11.933 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:16:11.933 INFO:tasks.workunit.client.0.smithi172.stdout:7 2024-05-08T17:16:11.934 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-05-08T17:16:11.934 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:16:11.934 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T17:16:12.464 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:16:13.465 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:16:13.466 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:16:13.466 INFO:tasks.workunit.client.0.smithi172.stdout:8 2024-05-08T17:16:13.467 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-05-08T17:16:13.467 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:16:13.467 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T17:16:13.902 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:16:14.904 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:16:14.905 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:16:14.905 INFO:tasks.workunit.client.0.smithi172.stdout:9 2024-05-08T17:16:14.905 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-05-08T17:16:14.905 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:16:14.906 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T17:16:15.827 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:16:16.828 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:16:16.829 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:16:16.829 INFO:tasks.workunit.client.0.smithi172.stdout:10 2024-05-08T17:16:16.829 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-05-08T17:16:16.830 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:16:16.830 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T17:16:17.902 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:16:18.904 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:16:18.904 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:16:18.904 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-05-08T17:16:18.904 INFO:tasks.workunit.client.0.smithi172.stdout:11 2024-05-08T17:16:18.905 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:16:18.905 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T17:16:19.941 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:16:20.942 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:16:20.943 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:16:20.943 INFO:tasks.workunit.client.0.smithi172.stdout:12 2024-05-08T17:16:20.943 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-05-08T17:16:20.944 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:16:20.944 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T17:16:22.000 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:16:23.002 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:16:23.002 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:16:23.003 INFO:tasks.workunit.client.0.smithi172.stdout:13 2024-05-08T17:16:23.003 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-05-08T17:16:23.003 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:16:23.003 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T17:16:23.716 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:16:24.718 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:16:24.718 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:16:24.718 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-05-08T17:16:24.719 INFO:tasks.workunit.client.0.smithi172.stdout:14 2024-05-08T17:16:24.719 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:16:24.719 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T17:16:25.348 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:16:26.350 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:16:26.350 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:16:26.350 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-05-08T17:16:26.350 INFO:tasks.workunit.client.0.smithi172.stdout:15 2024-05-08T17:16:26.351 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:16:26.351 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T17:16:26.825 INFO:tasks.workunit.client.0.smithi172.stdout:osd.2 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/2776502801,v1:127.0.0.1:6819/2776502801] [v2:127.0.0.1:6820/2776502801,v1:127.0.0.1:6821/2776502801] exists,up 383cfd0c-3d6e-43a7-8b7c-bcd89e0b2c03 2024-05-08T17:16:26.829 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-08T17:16:26.829 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-08T17:16:26.829 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-08T17:16:26.829 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-05-08T17:16:26.829 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 3 2024-05-08T17:16:26.829 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-05-08T17:16:26.829 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-05-08T17:16:26.829 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2024-05-08T17:16:26.829 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-05-08T17:16:26.829 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/3 2024-05-08T17:16:26.829 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=4457b0f1-028b-482b-8ee3-7ee05861e6cc --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-05-08T17:16:26.830 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-08T17:16:26.830 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-05-08T17:16:26.830 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-08T17:16:26.830 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/3' 2024-05-08T17:16:26.830 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/3/journal' 2024-05-08T17:16:26.830 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-05-08T17:16:26.830 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-05-08T17:16:26.830 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-05-08T17:16:26.830 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-05-08T17:16:26.830 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-08T17:16:26.830 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-08T17:16:26.830 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-08T17:16:26.831 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T17:16:26.831 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81110 2024-05-08T17:16:26.831 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.81110/$cluster-$name.asok' 2024-05-08T17:16:26.831 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.81110/$cluster-$name.asok' 2024-05-08T17:16:26.831 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-05-08T17:16:26.831 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-05-08T17:16:26.831 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-05-08T17:16:26.831 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-05-08T17:16:26.831 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-05-08T17:16:26.831 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-08T17:16:26.831 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-08T17:16:26.831 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-08T17:16:26.831 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-08T17:16:26.831 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-05-08T17:16:26.831 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-05-08T17:16:26.832 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/3 2024-05-08T17:16:26.832 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-05-08T17:16:26.833 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=29e46a90-d974-4d60-8b89-dff771977e7f 2024-05-08T17:16:26.833 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 29e46a90-d974-4d60-8b89-dff771977e7f' 2024-05-08T17:16:26.833 INFO:tasks.workunit.client.0.smithi172.stdout:add osd3 29e46a90-d974-4d60-8b89-dff771977e7f 2024-05-08T17:16:26.833 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-05-08T17:16:26.853 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBqsztmRJwwMxAABgEVfnAevQVadjW4cYCc1g== 2024-05-08T17:16:26.853 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBqsztmRJwwMxAABgEVfnAevQVadjW4cYCc1g=="}' 2024-05-08T17:16:26.853 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 29e46a90-d974-4d60-8b89-dff771977e7f -i td/test-erasure-eio/3/new.json 2024-05-08T17:16:27.573 INFO:tasks.workunit.client.0.smithi172.stdout:3 2024-05-08T17:16:27.596 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/3/new.json 2024-05-08T17:16:27.598 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=4457b0f1-028b-482b-8ee3-7ee05861e6cc --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81110/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBqsztmRJwwMxAABgEVfnAevQVadjW4cYCc1g== --osd-uuid 29e46a90-d974-4d60-8b89-dff771977e7f 2024-05-08T17:16:27.628 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:16:27.633+0000 7f27f8b207c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T17:16:27.677 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:16:27.681+0000 7f27f8b207c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T17:16:27.679 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:16:27.681+0000 7f27f8b207c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T17:16:27.679 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:16:27.681+0000 7f27f8b207c0 -1 bluestore(td/test-erasure-eio/3/block) _read_bdev_label failed to open td/test-erasure-eio/3/block: (2) No such file or directory 2024-05-08T17:16:27.679 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:16:27.681+0000 7f27f8b207c0 -1 bluestore(td/test-erasure-eio/3/block) _read_bdev_label failed to open td/test-erasure-eio/3/block: (2) No such file or directory 2024-05-08T17:16:27.679 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:16:27.681+0000 7f27f8b207c0 -1 bluestore(td/test-erasure-eio/3/block) _read_bdev_label failed to open td/test-erasure-eio/3/block: (2) No such file or directory 2024-05-08T17:16:27.812 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:16:27.817+0000 7f27f8b207c0 -1 bluestore(td/test-erasure-eio/3) _read_fsid unparsable uuid 2024-05-08T17:16:34.898 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/3/keyring 2024-05-08T17:16:34.898 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-05-08T17:16:34.899 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2024-05-08T17:16:34.899 INFO:tasks.workunit.client.0.smithi172.stdout:adding osd3 key to auth repository 2024-05-08T17:16:34.899 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-05-08T17:16:35.446 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2024-05-08T17:16:35.447 INFO:tasks.workunit.client.0.smithi172.stdout:start osd.3 2024-05-08T17:16:35.447 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=4457b0f1-028b-482b-8ee3-7ee05861e6cc --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81110/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-05-08T17:16:35.447 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-05-08T17:16:35.447 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-05-08T17:16:35.447 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-05-08T17:16:35.480 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:16:35.480+0000 7ff39ded27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T17:16:35.482 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:16:35.484+0000 7ff39ded27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T17:16:35.483 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:16:35.484+0000 7ff39ded27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T17:16:35.925 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2024-05-08T17:16:35.925 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-08T17:16:35.925 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-05-08T17:16:35.925 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-08T17:16:35.925 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-08T17:16:35.925 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:16:35.925 INFO:tasks.workunit.client.0.smithi172.stdout:0 2024-05-08T17:16:35.926 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-08T17:16:35.926 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T17:16:35.926 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:16:36.033 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:16:36.036+0000 7ff39ded27c0 -1 Falling back to public interface 2024-05-08T17:16:36.354 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:16:37.355 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:16:37.356 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:16:37.356 INFO:tasks.workunit.client.0.smithi172.stdout:1 2024-05-08T17:16:37.356 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-08T17:16:37.356 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:16:37.357 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T17:16:37.771 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:16:38.679 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:16:38.680+0000 7ff39ded27c0 -1 osd.3 0 log_to_monitors true 2024-05-08T17:16:38.773 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:16:38.773 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:16:38.773 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-08T17:16:38.774 INFO:tasks.workunit.client.0.smithi172.stdout:2 2024-05-08T17:16:38.775 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:16:38.775 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T17:16:39.422 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:16:40.424 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:16:40.424 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:16:40.424 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-05-08T17:16:40.424 INFO:tasks.workunit.client.0.smithi172.stdout:3 2024-05-08T17:16:40.425 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:16:40.425 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T17:16:41.216 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:16:42.218 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:16:42.218 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:16:42.218 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-05-08T17:16:42.218 INFO:tasks.workunit.client.0.smithi172.stdout:4 2024-05-08T17:16:42.219 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:16:42.219 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T17:16:42.670 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:16:43.672 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:16:43.673 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:16:43.673 INFO:tasks.workunit.client.0.smithi172.stdout:5 2024-05-08T17:16:43.674 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-05-08T17:16:43.674 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:16:43.675 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T17:16:44.119 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:16:45.121 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:16:45.121 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:16:45.121 INFO:tasks.workunit.client.0.smithi172.stdout:6 2024-05-08T17:16:45.122 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-05-08T17:16:45.122 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:16:45.122 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T17:16:45.631 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:16:46.633 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:16:46.634 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:16:46.634 INFO:tasks.workunit.client.0.smithi172.stdout:7 2024-05-08T17:16:46.635 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-05-08T17:16:46.635 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:16:46.635 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T17:16:47.096 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:16:48.097 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:16:48.098 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:16:48.098 INFO:tasks.workunit.client.0.smithi172.stdout:8 2024-05-08T17:16:48.099 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-05-08T17:16:48.099 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:16:48.099 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T17:16:48.611 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:16:49.613 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:16:49.613 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:16:49.613 INFO:tasks.workunit.client.0.smithi172.stdout:9 2024-05-08T17:16:49.614 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-05-08T17:16:49.614 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:16:49.614 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T17:16:50.077 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:16:51.079 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:16:51.079 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:16:51.079 INFO:tasks.workunit.client.0.smithi172.stdout:10 2024-05-08T17:16:51.080 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-05-08T17:16:51.081 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:16:51.081 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T17:16:51.691 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:16:52.693 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:16:52.693 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:16:52.693 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-05-08T17:16:52.693 INFO:tasks.workunit.client.0.smithi172.stdout:11 2024-05-08T17:16:52.694 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:16:52.694 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T17:16:53.308 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:16:54.310 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:16:54.310 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:16:54.310 INFO:tasks.workunit.client.0.smithi172.stdout:12 2024-05-08T17:16:54.311 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-05-08T17:16:54.311 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:16:54.311 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T17:16:55.034 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:16:56.039 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:16:56.039 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:16:56.039 INFO:tasks.workunit.client.0.smithi172.stdout:13 2024-05-08T17:16:56.040 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-05-08T17:16:56.040 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:16:56.040 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T17:16:57.089 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:16:58.091 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:16:58.091 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:16:58.091 INFO:tasks.workunit.client.0.smithi172.stdout:14 2024-05-08T17:16:58.092 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-05-08T17:16:58.092 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:16:58.092 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T17:16:59.237 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:17:00.239 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:17:00.239 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:17:00.240 INFO:tasks.workunit.client.0.smithi172.stdout:15 2024-05-08T17:17:00.240 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-05-08T17:17:00.240 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:17:00.240 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T17:17:01.205 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:17:02.206 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:17:02.206 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:17:02.207 INFO:tasks.workunit.client.0.smithi172.stdout:16 2024-05-08T17:17:02.207 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-05-08T17:17:02.207 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:17:02.207 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T17:17:03.335 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:17:04.337 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:17:04.337 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:17:04.337 INFO:tasks.workunit.client.0.smithi172.stdout:17 2024-05-08T17:17:04.337 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-05-08T17:17:04.338 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:17:04.338 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T17:17:05.406 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:17:06.408 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:17:06.408 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:17:06.408 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-05-08T17:17:06.408 INFO:tasks.workunit.client.0.smithi172.stdout:18 2024-05-08T17:17:06.409 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:17:06.409 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T17:17:07.095 INFO:tasks.workunit.client.0.smithi172.stdout:osd.3 up in weight 1 up_from 30 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/3464296557,v1:127.0.0.1:6827/3464296557] [v2:127.0.0.1:6828/3464296557,v1:127.0.0.1:6829/3464296557] exists,up 29e46a90-d974-4d60-8b89-dff771977e7f 2024-05-08T17:17:07.095 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-08T17:17:07.096 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-08T17:17:07.096 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-08T17:17:07.096 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: get_asok_path osd.0 2024-05-08T17:17:07.096 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-05-08T17:17:07.096 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-05-08T17:17:07.097 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-08T17:17:07.097 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T17:17:07.097 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81110 2024-05-08T17:17:07.097 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.81110/ceph-osd.0.asok 2024-05-08T17:17:07.097 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: CEPH_ARGS= 2024-05-08T17:17:07.097 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: ceph --admin-daemon /tmp/ceph-asok.81110/ceph-osd.0.asok log flush 2024-05-08T17:17:07.208 INFO:tasks.workunit.client.0.smithi172.stdout:{} 2024-05-08T17:17:07.221 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:56: setup_osds: grep 'load: jerasure.*lrc' td/test-erasure-eio/osd.0.log 2024-05-08T17:17:07.226 INFO:tasks.workunit.client.0.smithi172.stdout:2024-05-08T17:14:54.619+0000 7efdd79207c0 0 load: jerasure load: lrc 2024-05-08T17:17:07.227 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:335: TEST_rados_get_with_subreadall_eio_shard_0: local poolname=pool-jerasure 2024-05-08T17:17:07.228 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:336: TEST_rados_get_with_subreadall_eio_shard_0: create_erasure_coded_pool pool-jerasure 2 1 2024-05-08T17:17:07.228 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:67: create_erasure_coded_pool: local poolname=pool-jerasure 2024-05-08T17:17:07.228 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:68: create_erasure_coded_pool: shift 2024-05-08T17:17:07.228 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:69: create_erasure_coded_pool: local k=2 2024-05-08T17:17:07.228 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:70: create_erasure_coded_pool: shift 2024-05-08T17:17:07.228 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:71: create_erasure_coded_pool: local m=1 2024-05-08T17:17:07.228 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:72: create_erasure_coded_pool: shift 2024-05-08T17:17:07.228 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:74: create_erasure_coded_pool: ceph osd erasure-code-profile set myprofile plugin=jerasure k=2 m=1 crush-failure-domain=osd 2024-05-08T17:17:07.953 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:78: create_erasure_coded_pool: create_pool pool-jerasure 1 1 erasure myprofile 2024-05-08T17:17:07.953 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create pool-jerasure 1 1 erasure myprofile 2024-05-08T17:17:09.189 INFO:tasks.workunit.client.0.smithi172.stderr:pool 'pool-jerasure' already exists 2024-05-08T17:17:09.206 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-05-08T17:17:10.208 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:80: create_erasure_coded_pool: wait_for_clean 2024-05-08T17:17:10.209 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-05-08T17:17:10.209 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-05-08T17:17:10.209 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-05-08T17:17:10.210 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-05-08T17:17:10.210 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-05-08T17:17:10.210 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-05-08T17:17:10.211 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-05-08T17:17:10.212 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-05-08T17:17:10.212 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-05-08T17:17:10.338 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-05-08T17:17:10.338 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-05-08T17:17:10.338 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-05-08T17:17:10.338 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-05-08T17:17:10.338 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-05-08T17:17:10.339 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-05-08T17:17:10.830 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-05-08T17:17:10.831 INFO:tasks.workunit.client.0.smithi172.stderr:1 2024-05-08T17:17:10.831 INFO:tasks.workunit.client.0.smithi172.stderr:2 2024-05-08T17:17:10.831 INFO:tasks.workunit.client.0.smithi172.stderr:3' 2024-05-08T17:17:10.831 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-05-08T17:17:10.831 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T17:17:10.831 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-05-08T17:17:10.993 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=25769803802 2024-05-08T17:17:10.993 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 25769803802 2024-05-08T17:17:10.993 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803802' 2024-05-08T17:17:10.993 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T17:17:10.993 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-05-08T17:17:11.145 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=55834574874 2024-05-08T17:17:11.145 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 55834574874 2024-05-08T17:17:11.145 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803802 1-55834574874' 2024-05-08T17:17:11.145 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T17:17:11.145 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-05-08T17:17:11.295 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=85899345936 2024-05-08T17:17:11.295 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 85899345936 2024-05-08T17:17:11.295 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803802 1-55834574874 2-85899345936' 2024-05-08T17:17:11.296 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T17:17:11.296 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-05-08T17:17:11.446 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=128849018887 2024-05-08T17:17:11.446 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 128849018887 2024-05-08T17:17:11.446 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803802 1-55834574874 2-85899345936 3-128849018887' 2024-05-08T17:17:11.446 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-08T17:17:11.447 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-25769803802 2024-05-08T17:17:11.447 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-08T17:17:11.448 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-05-08T17:17:11.449 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-25769803802 2024-05-08T17:17:11.449 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-08T17:17:11.451 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=25769803802 2024-05-08T17:17:11.451 INFO:tasks.workunit.client.0.smithi172.stdout:waiting osd.0 seq 25769803802 2024-05-08T17:17:11.452 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 25769803802' 2024-05-08T17:17:11.452 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-05-08T17:17:11.888 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 25769803799 -lt 25769803802 2024-05-08T17:17:11.889 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-05-08T17:17:12.890 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-05-08T17:17:12.891 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-05-08T17:17:13.336 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 25769803802 -lt 25769803802 2024-05-08T17:17:13.336 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-08T17:17:13.337 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-55834574874 2024-05-08T17:17:13.337 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-08T17:17:13.338 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-05-08T17:17:13.339 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-55834574874 2024-05-08T17:17:13.340 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-08T17:17:13.341 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=55834574874 2024-05-08T17:17:13.341 INFO:tasks.workunit.client.0.smithi172.stdout:waiting osd.1 seq 55834574874 2024-05-08T17:17:13.341 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 55834574874' 2024-05-08T17:17:13.342 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-05-08T17:17:13.747 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 55834574874 -lt 55834574874 2024-05-08T17:17:13.747 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-08T17:17:13.747 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-85899345936 2024-05-08T17:17:13.748 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-08T17:17:13.749 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-05-08T17:17:13.750 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-85899345936 2024-05-08T17:17:13.750 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-08T17:17:13.752 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=85899345936 2024-05-08T17:17:13.752 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 85899345936' 2024-05-08T17:17:13.752 INFO:tasks.workunit.client.0.smithi172.stdout:waiting osd.2 seq 85899345936 2024-05-08T17:17:13.752 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-05-08T17:17:14.171 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 85899345937 -lt 85899345936 2024-05-08T17:17:14.171 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-08T17:17:14.171 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-128849018887 2024-05-08T17:17:14.172 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-08T17:17:14.173 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-05-08T17:17:14.174 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-128849018887 2024-05-08T17:17:14.174 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-08T17:17:14.176 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=128849018887 2024-05-08T17:17:14.176 INFO:tasks.workunit.client.0.smithi172.stdout:waiting osd.3 seq 128849018887 2024-05-08T17:17:14.176 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 128849018887' 2024-05-08T17:17:14.176 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-05-08T17:17:14.619 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 128849018887 -lt 128849018887 2024-05-08T17:17:14.620 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-05-08T17:17:14.620 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-05-08T17:17:14.620 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-05-08T17:17:15.156 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-05-08T17:17:15.156 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-05-08T17:17:15.156 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-05-08T17:17:15.156 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-05-08T17:17:15.156 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-05-08T17:17:15.156 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-05-08T17:17:15.156 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-05-08T17:17:15.156 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-05-08T17:17:15.582 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-05-08T17:17:15.583 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-05-08T17:17:15.583 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-05-08T17:17:15.583 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-05-08T17:17:16.113 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-05-08T17:17:16.113 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-05-08T17:17:16.113 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-05-08T17:17:16.113 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:338: TEST_rados_get_with_subreadall_eio_shard_0: rados_put_get_data eio td/test-erasure-eio 0 recovery 2024-05-08T17:17:16.113 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:147: rados_put_get_data: local inject=eio 2024-05-08T17:17:16.113 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:148: rados_put_get_data: shift 2024-05-08T17:17:16.114 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:149: rados_put_get_data: local dir=td/test-erasure-eio 2024-05-08T17:17:16.114 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:150: rados_put_get_data: shift 2024-05-08T17:17:16.114 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:151: rados_put_get_data: local shard_id=0 2024-05-08T17:17:16.114 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:152: rados_put_get_data: shift 2024-05-08T17:17:16.114 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:153: rados_put_get_data: local arg=recovery 2024-05-08T17:17:16.114 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:157: rados_put_get_data: local poolname=pool-jerasure 2024-05-08T17:17:16.114 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:158: rados_put_get_data: local objname=obj-eio-81110-0 2024-05-08T17:17:16.114 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:159: rados_put_get_data: rados_put td/test-erasure-eio pool-jerasure obj-eio-81110-0 2024-05-08T17:17:16.114 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:90: rados_put: local dir=td/test-erasure-eio 2024-05-08T17:17:16.114 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:91: rados_put: local poolname=pool-jerasure 2024-05-08T17:17:16.114 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:92: rados_put: local objname=obj-eio-81110-0 2024-05-08T17:17:16.114 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-05-08T17:17:16.114 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 AAA 2024-05-08T17:17:16.115 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-05-08T17:17:16.115 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 BBB 2024-05-08T17:17:16.115 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-05-08T17:17:16.115 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 CCCC 2024-05-08T17:17:16.115 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-05-08T17:17:16.115 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 DDDD 2024-05-08T17:17:16.115 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:100: rados_put: rados --pool pool-jerasure put obj-eio-81110-0 td/test-erasure-eio/ORIGINAL 2024-05-08T17:17:16.290 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:160: rados_put_get_data: inject_eio ec data pool-jerasure obj-eio-81110-0 td/test-erasure-eio 0 2024-05-08T17:17:16.291 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2394: inject_eio: local pooltype=ec 2024-05-08T17:17:16.291 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2395: inject_eio: shift 2024-05-08T17:17:16.291 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2396: inject_eio: local which=data 2024-05-08T17:17:16.291 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2397: inject_eio: shift 2024-05-08T17:17:16.291 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2398: inject_eio: local poolname=pool-jerasure 2024-05-08T17:17:16.291 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2399: inject_eio: shift 2024-05-08T17:17:16.291 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2400: inject_eio: local objname=obj-eio-81110-0 2024-05-08T17:17:16.291 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2401: inject_eio: shift 2024-05-08T17:17:16.291 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2402: inject_eio: local dir=td/test-erasure-eio 2024-05-08T17:17:16.291 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2403: inject_eio: shift 2024-05-08T17:17:16.291 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2404: inject_eio: local shard_id=0 2024-05-08T17:17:16.291 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2405: inject_eio: shift 2024-05-08T17:17:16.292 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: get_osds pool-jerasure obj-eio-81110-0 2024-05-08T17:17:16.292 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2024-05-08T17:17:16.292 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-eio-81110-0 2024-05-08T17:17:16.292 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure obj-eio-81110-0 2024-05-08T17:17:16.293 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-05-08T17:17:16.720 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-05-08T17:17:16.790 INFO:tasks.workunit.client.0.smithi172.stderr:1 2024-05-08T17:17:16.790 INFO:tasks.workunit.client.0.smithi172.stderr:2' 2024-05-08T17:17:16.790 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 2 2024-05-08T17:17:16.790 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: initial_osds=('3' '1' '2') 2024-05-08T17:17:16.790 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: local -a initial_osds 2024-05-08T17:17:16.791 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2408: inject_eio: local osd_id=3 2024-05-08T17:17:16.791 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2409: inject_eio: '[' ec '!=' ec ']' 2024-05-08T17:17:16.793 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: cat td/test-erasure-eio/3/type 2024-05-08T17:17:16.793 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: type=bluestore 2024-05-08T17:17:16.793 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: set_config osd 3 bluestore_debug_inject_read_err true 2024-05-08T17:17:16.793 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-05-08T17:17:16.793 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-05-08T17:17:16.793 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-05-08T17:17:16.793 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-05-08T17:17:16.793 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-05-08T17:17:16.793 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-05-08T17:17:16.793 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-05-08T17:17:16.793 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-05-08T17:17:16.794 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-08T17:17:16.794 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T17:17:16.794 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81110 2024-05-08T17:17:16.794 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.81110/ceph-osd.3.asok 2024-05-08T17:17:16.794 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.81110/ceph-osd.3.asok config set bluestore_debug_inject_read_err true 2024-05-08T17:17:16.832 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-05-08T17:17:16.832 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local loop=0 2024-05-08T17:17:16.834 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: grep -q Invalid 2024-05-08T17:17:16.834 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: get_asok_path osd.3 2024-05-08T17:17:16.834 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-05-08T17:17:16.834 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-05-08T17:17:16.835 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-08T17:17:16.835 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T17:17:16.835 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81110 2024-05-08T17:17:16.835 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.81110/ceph-osd.3.asok 2024-05-08T17:17:16.835 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: CEPH_ARGS= 2024-05-08T17:17:16.836 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: ceph --admin-daemon /tmp/ceph-asok.81110/ceph-osd.3.asok injectdataerr pool-jerasure obj-eio-81110-0 0 2024-05-08T17:17:16.945 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:161: rados_put_get_data: rados_get td/test-erasure-eio pool-jerasure obj-eio-81110-0 2024-05-08T17:17:16.946 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2024-05-08T17:17:16.946 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2024-05-08T17:17:16.946 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=obj-eio-81110-0 2024-05-08T17:17:16.946 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=ok 2024-05-08T17:17:16.946 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' ok = fail ']' 2024-05-08T17:17:16.946 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:120: rados_get: rados --pool pool-jerasure get obj-eio-81110-0 td/test-erasure-eio/COPY 2024-05-08T17:17:16.983 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:121: rados_get: diff td/test-erasure-eio/ORIGINAL td/test-erasure-eio/COPY 2024-05-08T17:17:16.983 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:122: rados_get: rm td/test-erasure-eio/COPY 2024-05-08T17:17:16.984 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:163: rados_put_get_data: '[' recovery = recovery ']' 2024-05-08T17:17:16.985 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:170: rados_put_get_data: get_osds pool-jerasure obj-eio-81110-0 2024-05-08T17:17:16.985 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2024-05-08T17:17:16.985 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-eio-81110-0 2024-05-08T17:17:16.985 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure obj-eio-81110-0 2024-05-08T17:17:16.985 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-05-08T17:17:17.374 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-05-08T17:17:17.374 INFO:tasks.workunit.client.0.smithi172.stderr:1 2024-05-08T17:17:17.374 INFO:tasks.workunit.client.0.smithi172.stderr:2' 2024-05-08T17:17:17.374 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 2 2024-05-08T17:17:17.375 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:170: rados_put_get_data: initial_osds=('3' '1' '2') 2024-05-08T17:17:17.375 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:170: rados_put_get_data: local -a initial_osds 2024-05-08T17:17:17.375 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:171: rados_put_get_data: local last_osd=2 2024-05-08T17:17:17.375 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:173: rados_put_get_data: kill_daemons td/test-erasure-eio TERM osd.2 2024-05-08T17:17:17.375 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-08T17:17:17.375 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-08T17:17:17.375 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-08T17:17:17.375 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-08T17:17:17.376 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-08T17:17:17.683 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-08T17:17:17.683 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:174: rados_put_get_data: ceph osd out 2 2024-05-08T17:17:18.851 INFO:tasks.workunit.client.0.smithi172.stderr:osd.2 is already out. 2024-05-08T17:17:18.871 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:175: rados_put_get_data: get_osds pool-jerasure obj-eio-81110-0 2024-05-08T17:17:18.871 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:175: rados_put_get_data: grep '\<2\>' 2024-05-08T17:17:18.872 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2024-05-08T17:17:18.872 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-eio-81110-0 2024-05-08T17:17:18.872 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure obj-eio-81110-0 2024-05-08T17:17:18.872 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-05-08T17:17:19.370 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-05-08T17:17:19.370 INFO:tasks.workunit.client.0.smithi172.stderr:1 2024-05-08T17:17:19.370 INFO:tasks.workunit.client.0.smithi172.stderr:0' 2024-05-08T17:17:19.370 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-05-08T17:17:19.370 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:176: rados_put_get_data: ceph osd in 2 2024-05-08T17:17:20.038 INFO:tasks.workunit.client.0.smithi172.stderr:osd.2 is already in. 2024-05-08T17:17:20.057 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:177: rados_put_get_data: activate_osd td/test-erasure-eio 2 2024-05-08T17:17:20.057 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/test-erasure-eio 2024-05-08T17:17:20.057 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-05-08T17:17:20.057 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-05-08T17:17:20.057 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-05-08T17:17:20.058 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/test-erasure-eio/2 2024-05-08T17:17:20.058 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=4457b0f1-028b-482b-8ee3-7ee05861e6cc --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-05-08T17:17:20.058 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-08T17:17:20.058 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-05-08T17:17:20.058 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-08T17:17:20.058 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/2' 2024-05-08T17:17:20.058 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/2/journal' 2024-05-08T17:17:20.058 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-05-08T17:17:20.058 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-05-08T17:17:20.058 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-05-08T17:17:20.059 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-05-08T17:17:20.059 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-08T17:17:20.059 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-08T17:17:20.059 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-08T17:17:20.059 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T17:17:20.059 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81110 2024-05-08T17:17:20.060 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.81110/$cluster-$name.asok' 2024-05-08T17:17:20.060 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.81110/$cluster-$name.asok' 2024-05-08T17:17:20.060 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-05-08T17:17:20.061 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-05-08T17:17:20.061 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-05-08T17:17:20.061 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-08T17:17:20.061 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-08T17:17:20.061 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-08T17:17:20.061 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-08T17:17:20.061 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-05-08T17:17:20.061 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-05-08T17:17:20.061 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/test-erasure-eio/2 2024-05-08T17:17:20.062 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-05-08T17:17:20.062 INFO:tasks.workunit.client.0.smithi172.stdout:start osd.2 2024-05-08T17:17:20.063 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=4457b0f1-028b-482b-8ee3-7ee05861e6cc --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81110/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-05-08T17:17:20.063 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/test-erasure-eio/2/whoami 2024-05-08T17:17:20.063 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-05-08T17:17:20.063 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-05-08T17:17:20.063 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-05-08T17:17:20.063 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-05-08T17:17:20.087 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:17:20.087+0000 7f806fd897c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T17:17:20.089 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:17:20.091+0000 7f806fd897c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T17:17:20.091 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:17:20.091+0000 7f806fd897c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T17:17:20.629 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-05-08T17:17:20.630 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-08T17:17:20.630 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-05-08T17:17:20.630 INFO:tasks.workunit.client.0.smithi172.stdout:0 2024-05-08T17:17:20.631 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-08T17:17:20.631 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-08T17:17:20.631 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:17:20.631 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-08T17:17:20.631 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:17:20.631 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T17:17:20.653 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:17:20.655+0000 7f806fd897c0 -1 Falling back to public interface 2024-05-08T17:17:21.353 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:17:22.355 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:17:22.355 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:17:22.355 INFO:tasks.workunit.client.0.smithi172.stdout:1 2024-05-08T17:17:22.356 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-08T17:17:22.356 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:17:22.356 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T17:17:22.796 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:17:23.798 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:17:23.798 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:17:23.799 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-08T17:17:23.799 INFO:tasks.workunit.client.0.smithi172.stdout:2 2024-05-08T17:17:23.799 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:17:23.800 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T17:17:23.858 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:17:23.859+0000 7f806fd897c0 -1 osd.2 39 log_to_monitors true 2024-05-08T17:17:24.268 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:17:25.269 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:17:25.270 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:17:25.270 INFO:tasks.workunit.client.0.smithi172.stdout:3 2024-05-08T17:17:25.270 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-05-08T17:17:25.271 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:17:25.271 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T17:17:25.905 INFO:tasks.workunit.client.0.smithi172.stdout:osd.2 up in weight 1 up_from 51 up_thru 21 down_at 40 last_clean_interval [20,39) [v2:127.0.0.1:6818/3013801308,v1:127.0.0.1:6819/3013801308] [v2:127.0.0.1:6820/3013801308,v1:127.0.0.1:6821/3013801308] exists,up 383cfd0c-3d6e-43a7-8b7c-bcd89e0b2c03 2024-05-08T17:17:25.907 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-08T17:17:25.908 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-08T17:17:25.908 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-08T17:17:25.908 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:178: rados_put_get_data: wait_for_clean 2024-05-08T17:17:25.908 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-05-08T17:17:25.908 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-05-08T17:17:25.908 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-05-08T17:17:25.908 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-05-08T17:17:25.908 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-05-08T17:17:25.908 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-05-08T17:17:25.908 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-05-08T17:17:25.909 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-05-08T17:17:25.909 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-05-08T17:17:26.027 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-05-08T17:17:26.027 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-05-08T17:17:26.027 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-05-08T17:17:26.028 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-05-08T17:17:26.028 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-05-08T17:17:26.028 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-05-08T17:17:26.616 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-05-08T17:17:26.616 INFO:tasks.workunit.client.0.smithi172.stderr:1 2024-05-08T17:17:26.617 INFO:tasks.workunit.client.0.smithi172.stderr:2 2024-05-08T17:17:26.617 INFO:tasks.workunit.client.0.smithi172.stderr:3' 2024-05-08T17:17:26.617 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-05-08T17:17:26.617 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T17:17:26.617 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-05-08T17:17:26.789 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=25769803807 2024-05-08T17:17:26.789 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 25769803807 2024-05-08T17:17:26.790 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803807' 2024-05-08T17:17:26.790 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T17:17:26.790 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-05-08T17:17:27.111 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=55834574879 2024-05-08T17:17:27.112 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 55834574879 2024-05-08T17:17:27.112 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803807 1-55834574879' 2024-05-08T17:17:27.112 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T17:17:27.112 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-05-08T17:17:27.281 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=219043332099 2024-05-08T17:17:27.281 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 219043332099 2024-05-08T17:17:27.282 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803807 1-55834574879 2-219043332099' 2024-05-08T17:17:27.282 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T17:17:27.282 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-05-08T17:17:27.431 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=128849018892 2024-05-08T17:17:27.432 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 128849018892 2024-05-08T17:17:27.432 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803807 1-55834574879 2-219043332099 3-128849018892' 2024-05-08T17:17:27.432 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-08T17:17:27.432 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-25769803807 2024-05-08T17:17:27.432 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-08T17:17:27.434 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-05-08T17:17:27.435 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-08T17:17:27.435 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-25769803807 2024-05-08T17:17:27.436 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=25769803807 2024-05-08T17:17:27.436 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 25769803807' 2024-05-08T17:17:27.436 INFO:tasks.workunit.client.0.smithi172.stdout:waiting osd.0 seq 25769803807 2024-05-08T17:17:27.436 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-05-08T17:17:27.829 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 25769803805 -lt 25769803807 2024-05-08T17:17:27.829 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-05-08T17:17:28.831 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-05-08T17:17:28.831 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-05-08T17:17:29.295 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 25769803807 -lt 25769803807 2024-05-08T17:17:29.295 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-08T17:17:29.296 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-08T17:17:29.296 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-55834574879 2024-05-08T17:17:29.297 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-05-08T17:17:29.298 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-55834574879 2024-05-08T17:17:29.298 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-08T17:17:29.299 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=55834574879 2024-05-08T17:17:29.299 INFO:tasks.workunit.client.0.smithi172.stdout:waiting osd.1 seq 55834574879 2024-05-08T17:17:29.300 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 55834574879' 2024-05-08T17:17:29.300 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-05-08T17:17:29.771 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 55834574879 -lt 55834574879 2024-05-08T17:17:29.771 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-08T17:17:29.771 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-219043332099 2024-05-08T17:17:29.772 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-08T17:17:29.773 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-05-08T17:17:29.774 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-219043332099 2024-05-08T17:17:29.774 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-08T17:17:29.775 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=219043332099 2024-05-08T17:17:29.776 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 219043332099' 2024-05-08T17:17:29.776 INFO:tasks.workunit.client.0.smithi172.stdout:waiting osd.2 seq 219043332099 2024-05-08T17:17:29.776 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-05-08T17:17:30.187 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 219043332099 -lt 219043332099 2024-05-08T17:17:30.187 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-08T17:17:30.187 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-08T17:17:30.187 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-128849018892 2024-05-08T17:17:30.188 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-05-08T17:17:30.189 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-128849018892 2024-05-08T17:17:30.189 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-08T17:17:30.189 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=128849018892 2024-05-08T17:17:30.190 INFO:tasks.workunit.client.0.smithi172.stdout:waiting osd.3 seq 128849018892 2024-05-08T17:17:30.190 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 128849018892' 2024-05-08T17:17:30.190 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-05-08T17:17:30.618 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 128849018892 -lt 128849018892 2024-05-08T17:17:30.619 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-05-08T17:17:30.619 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-05-08T17:17:30.619 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-05-08T17:17:31.173 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-05-08T17:17:31.173 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-05-08T17:17:31.174 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-05-08T17:17:31.174 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-05-08T17:17:31.174 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-05-08T17:17:31.174 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-05-08T17:17:31.175 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-05-08T17:17:31.175 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-05-08T17:17:31.612 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-05-08T17:17:31.613 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-05-08T17:17:31.613 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-05-08T17:17:31.613 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-05-08T17:17:32.147 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-05-08T17:17:32.147 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-05-08T17:17:32.147 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-05-08T17:17:32.147 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:340: TEST_rados_get_with_subreadall_eio_shard_0: delete_erasure_coded_pool pool-jerasure 2024-05-08T17:17:32.147 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:84: delete_erasure_coded_pool: local poolname=pool-jerasure 2024-05-08T17:17:32.147 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:85: delete_erasure_coded_pool: ceph osd pool delete pool-jerasure pool-jerasure --yes-i-really-really-mean-it 2024-05-08T17:17:32.658 INFO:tasks.workunit.client.0.smithi172.stderr:pool 'pool-jerasure' does not exist 2024-05-08T17:17:32.675 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:86: delete_erasure_coded_pool: ceph osd erasure-code-profile rm myprofile 2024-05-08T17:17:33.267 INFO:tasks.workunit.client.0.smithi172.stderr:erasure-code-profile myprofile does not exist 2024-05-08T17:17:33.285 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:42: run: teardown td/test-erasure-eio 2024-05-08T17:17:33.286 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-eio 2024-05-08T17:17:33.286 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-05-08T17:17:33.286 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-eio KILL 2024-05-08T17:17:33.286 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-08T17:17:33.286 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-08T17:17:33.287 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-08T17:17:33.287 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-08T17:17:33.287 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-08T17:17:33.599 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-08T17:17:33.599 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-05-08T17:17:33.601 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-05-08T17:17:33.601 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-05-08T17:17:33.602 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-05-08T17:17:33.603 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-05-08T17:17:33.603 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-05-08T17:17:33.604 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-08T17:17:33.604 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-05-08T17:17:33.605 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-05-08T17:17:33.605 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-08T17:17:33.607 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-05-08T17:17:33.608 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-05-08T17:17:33.608 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-eio 2024-05-08T17:17:33.634 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-05-08T17:17:33.634 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T17:17:33.634 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81110 2024-05-08T17:17:33.634 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.81110 2024-05-08T17:17:33.636 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-05-08T17:17:33.636 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-05-08T17:17:33.636 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:32: run: for func in $funcs 2024-05-08T17:17:33.636 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:33: run: setup td/test-erasure-eio 2024-05-08T17:17:33.636 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/test-erasure-eio 2024-05-08T17:17:33.637 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/test-erasure-eio 2024-05-08T17:17:33.637 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-eio 2024-05-08T17:17:33.637 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-05-08T17:17:33.637 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-eio KILL 2024-05-08T17:17:33.637 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-08T17:17:33.637 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-08T17:17:33.637 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-08T17:17:33.637 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-08T17:17:33.637 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-08T17:17:33.639 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-08T17:17:33.639 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-05-08T17:17:33.640 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-05-08T17:17:33.641 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-05-08T17:17:33.642 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-05-08T17:17:33.642 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-05-08T17:17:33.642 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-05-08T17:17:33.643 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-08T17:17:33.643 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-05-08T17:17:33.643 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-05-08T17:17:33.644 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-08T17:17:33.644 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-05-08T17:17:33.646 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-05-08T17:17:33.646 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-eio 2024-05-08T17:17:33.647 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-05-08T17:17:33.647 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T17:17:33.647 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81110 2024-05-08T17:17:33.647 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.81110 2024-05-08T17:17:33.648 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-05-08T17:17:33.648 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-05-08T17:17:33.648 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/test-erasure-eio 2024-05-08T17:17:33.650 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-05-08T17:17:33.650 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T17:17:33.650 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81110 2024-05-08T17:17:33.650 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.81110 2024-05-08T17:17:33.651 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-05-08T17:17:33.652 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-05-08T17:17:33.653 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-05-08T17:17:33.653 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/test-erasure-eio 1' TERM HUP INT 2024-05-08T17:17:33.653 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:34: run: run_mon td/test-erasure-eio a 2024-05-08T17:17:33.653 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/test-erasure-eio 2024-05-08T17:17:33.653 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-05-08T17:17:33.653 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-05-08T17:17:33.653 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-05-08T17:17:33.653 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/test-erasure-eio/a 2024-05-08T17:17:33.653 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-erasure-eio/a --run-dir=td/test-erasure-eio 2024-05-08T17:17:33.997 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-05-08T17:17:33.998 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-08T17:17:33.998 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-08T17:17:33.998 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-08T17:17:33.998 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T17:17:33.998 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81110 2024-05-08T17:17:33.998 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.81110/$cluster-$name.asok' 2024-05-08T17:17:33.998 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-erasure-eio/a '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.81110/$cluster-$name.asok' --mon-cluster-log-file=td/test-erasure-eio/log --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-05-08T17:17:34.189 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-05-08T17:17:34.190 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-05-08T17:17:34.190 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-05-08T17:17:34.190 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-05-08T17:17:34.190 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-05-08T17:17:34.191 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-05-08T17:17:34.191 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-05-08T17:17:34.191 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-05-08T17:17:34.192 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-05-08T17:17:34.192 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-08T17:17:34.192 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T17:17:34.192 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81110 2024-05-08T17:17:34.192 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.81110/ceph-mon.a.asok 2024-05-08T17:17:34.192 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-05-08T17:17:34.193 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.81110/ceph-mon.a.asok config get fsid 2024-05-08T17:17:34.304 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-05-08T17:17:34.304 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-05-08T17:17:34.304 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-05-08T17:17:34.304 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-05-08T17:17:34.305 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-05-08T17:17:34.305 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-05-08T17:17:34.305 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-05-08T17:17:34.305 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-05-08T17:17:34.306 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-08T17:17:34.306 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T17:17:34.306 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81110 2024-05-08T17:17:34.306 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.81110/ceph-mon.a.asok 2024-05-08T17:17:34.306 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-05-08T17:17:34.307 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.81110/ceph-mon.a.asok config get mon_host 2024-05-08T17:17:34.408 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:35: run: run_mgr td/test-erasure-eio x 2024-05-08T17:17:34.408 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/test-erasure-eio 2024-05-08T17:17:34.408 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-05-08T17:17:34.408 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-05-08T17:17:34.408 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-05-08T17:17:34.408 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/test-erasure-eio/x 2024-05-08T17:17:34.408 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-05-08T17:17:35.312 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-05-08T17:17:35.312 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-08T17:17:35.312 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-08T17:17:35.312 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-08T17:17:35.313 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T17:17:35.313 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81110 2024-05-08T17:17:35.313 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.81110/$cluster-$name.asok' 2024-05-08T17:17:35.313 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-05-08T17:17:35.314 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-erasure-eio/x '--log-file=td/test-erasure-eio/$name.log' '--admin-socket=/tmp/ceph-asok.81110/$cluster-$name.asok' --run-dir=td/test-erasure-eio '--pid-file=td/test-erasure-eio/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-05-08T17:17:35.347 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:36: run: create_pool rbd 4 2024-05-08T17:17:35.347 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-05-08T17:17:35.642 INFO:tasks.workunit.client.0.smithi172.stderr:pool 'rbd' already exists 2024-05-08T17:17:35.661 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-05-08T17:17:36.664 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: get_asok_path mon.a 2024-05-08T17:17:36.664 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-05-08T17:17:36.664 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-05-08T17:17:36.665 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-08T17:17:36.665 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T17:17:36.665 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81110 2024-05-08T17:17:36.666 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.81110/ceph-mon.a.asok 2024-05-08T17:17:36.666 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: CEPH_ARGS= 2024-05-08T17:17:36.666 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:39: run: ceph --admin-daemon /tmp/ceph-asok.81110/ceph-mon.a.asok log flush 2024-05-08T17:17:36.765 INFO:tasks.workunit.client.0.smithi172.stdout:{} 2024-05-08T17:17:36.777 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:40: run: grep 'load: jerasure.*lrc' td/test-erasure-eio/mon.a.log 2024-05-08T17:17:36.780 INFO:tasks.workunit.client.0.smithi172.stdout:2024-05-08T17:17:34.047+0000 7f59b24fbc80 0 load: jerasure load: lrc 2024-05-08T17:17:36.781 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:41: run: TEST_rados_get_with_subreadall_eio_shard_1 td/test-erasure-eio 2024-05-08T17:17:36.781 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:344: TEST_rados_get_with_subreadall_eio_shard_1: local dir=td/test-erasure-eio 2024-05-08T17:17:36.781 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:345: TEST_rados_get_with_subreadall_eio_shard_1: local shard_id=1 2024-05-08T17:17:36.781 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:347: TEST_rados_get_with_subreadall_eio_shard_1: setup_osds 4 2024-05-08T17:17:36.781 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:47: setup_osds: local count=4 2024-05-08T17:17:36.781 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:48: setup_osds: shift 2024-05-08T17:17:36.782 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: expr 4 - 1 2024-05-08T17:17:36.784 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: seq 0 3 2024-05-08T17:17:36.788 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-05-08T17:17:36.788 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 0 2024-05-08T17:17:36.788 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-05-08T17:17:36.789 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-05-08T17:17:36.789 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-05-08T17:17:36.789 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-05-08T17:17:36.789 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/0 2024-05-08T17:17:36.789 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=4457b0f1-028b-482b-8ee3-7ee05861e6cc --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-05-08T17:17:36.789 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-08T17:17:36.789 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-05-08T17:17:36.789 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-08T17:17:36.789 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/0' 2024-05-08T17:17:36.789 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/0/journal' 2024-05-08T17:17:36.789 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-05-08T17:17:36.789 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-05-08T17:17:36.789 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-05-08T17:17:36.789 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-05-08T17:17:36.789 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-08T17:17:36.790 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-08T17:17:36.790 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-08T17:17:36.790 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T17:17:36.790 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81110 2024-05-08T17:17:36.790 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.81110/$cluster-$name.asok' 2024-05-08T17:17:36.790 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.81110/$cluster-$name.asok' 2024-05-08T17:17:36.790 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-05-08T17:17:36.790 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-05-08T17:17:36.790 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-05-08T17:17:36.790 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-05-08T17:17:36.790 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-05-08T17:17:36.790 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-08T17:17:36.790 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-08T17:17:36.790 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-08T17:17:36.791 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-08T17:17:36.791 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-05-08T17:17:36.791 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-05-08T17:17:36.791 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/0 2024-05-08T17:17:36.791 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-05-08T17:17:36.791 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=2223a3e9-90e0-4788-a1fb-ce6b2f5035f8 2024-05-08T17:17:36.791 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 2223a3e9-90e0-4788-a1fb-ce6b2f5035f8' 2024-05-08T17:17:36.792 INFO:tasks.workunit.client.0.smithi172.stdout:add osd0 2223a3e9-90e0-4788-a1fb-ce6b2f5035f8 2024-05-08T17:17:36.792 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-05-08T17:17:36.807 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCwsztmYhF0MBAAvmKcROtZwNlYsYYBJN8qEw== 2024-05-08T17:17:36.807 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCwsztmYhF0MBAAvmKcROtZwNlYsYYBJN8qEw=="}' 2024-05-08T17:17:36.807 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 2223a3e9-90e0-4788-a1fb-ce6b2f5035f8 -i td/test-erasure-eio/0/new.json 2024-05-08T17:17:37.093 INFO:tasks.workunit.client.0.smithi172.stdout:0 2024-05-08T17:17:37.108 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/0/new.json 2024-05-08T17:17:37.109 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=4457b0f1-028b-482b-8ee3-7ee05861e6cc --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81110/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCwsztmYhF0MBAAvmKcROtZwNlYsYYBJN8qEw== --osd-uuid 2223a3e9-90e0-4788-a1fb-ce6b2f5035f8 2024-05-08T17:17:37.141 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:17:37.147+0000 7f73c51937c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T17:17:37.161 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:17:37.167+0000 7f73c51937c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T17:17:37.163 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:17:37.167+0000 7f73c51937c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T17:17:37.163 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:17:37.167+0000 7f73c51937c0 -1 bluestore(td/test-erasure-eio/0/block) _read_bdev_label failed to open td/test-erasure-eio/0/block: (2) No such file or directory 2024-05-08T17:17:37.163 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:17:37.167+0000 7f73c51937c0 -1 bluestore(td/test-erasure-eio/0/block) _read_bdev_label failed to open td/test-erasure-eio/0/block: (2) No such file or directory 2024-05-08T17:17:37.163 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:17:37.167+0000 7f73c51937c0 -1 bluestore(td/test-erasure-eio/0/block) _read_bdev_label failed to open td/test-erasure-eio/0/block: (2) No such file or directory 2024-05-08T17:17:37.201 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:17:37.207+0000 7f73c51937c0 -1 bluestore(td/test-erasure-eio/0) _read_fsid unparsable uuid 2024-05-08T17:17:42.919 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/0/keyring 2024-05-08T17:17:42.919 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-05-08T17:17:42.920 INFO:tasks.workunit.client.0.smithi172.stdout:adding osd0 key to auth repository 2024-05-08T17:17:42.921 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-05-08T17:17:42.921 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-05-08T17:17:43.484 INFO:tasks.workunit.client.0.smithi172.stdout:start osd.0 2024-05-08T17:17:43.484 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-05-08T17:17:43.484 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-05-08T17:17:43.485 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=4457b0f1-028b-482b-8ee3-7ee05861e6cc --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/0 --osd-journal=td/test-erasure-eio/0/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81110/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-05-08T17:17:43.485 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-05-08T17:17:43.485 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-05-08T17:17:43.515 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:17:43.519+0000 7f419bb7c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T17:17:43.517 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:17:43.523+0000 7f419bb7c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T17:17:43.518 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:17:43.523+0000 7f419bb7c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T17:17:43.929 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-05-08T17:17:43.929 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-08T17:17:43.929 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-05-08T17:17:43.929 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-08T17:17:43.929 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-08T17:17:43.929 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:17:43.929 INFO:tasks.workunit.client.0.smithi172.stdout:0 2024-05-08T17:17:43.930 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-08T17:17:43.930 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:17:43.930 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T17:17:44.081 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:17:44.087+0000 7f419bb7c7c0 -1 Falling back to public interface 2024-05-08T17:17:44.349 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:17:45.351 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:17:45.351 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:17:45.352 INFO:tasks.workunit.client.0.smithi172.stdout:1 2024-05-08T17:17:45.352 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-08T17:17:45.352 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:17:45.352 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T17:17:45.826 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:17:46.583 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:17:46.587+0000 7f419bb7c7c0 -1 osd.0 0 log_to_monitors true 2024-05-08T17:17:46.827 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:17:46.827 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:17:46.828 INFO:tasks.workunit.client.0.smithi172.stdout:2 2024-05-08T17:17:46.828 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-08T17:17:46.828 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:17:46.828 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T17:17:47.327 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:17:48.329 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:17:48.329 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:17:48.329 INFO:tasks.workunit.client.0.smithi172.stdout:3 2024-05-08T17:17:48.330 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-05-08T17:17:48.330 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:17:48.330 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T17:17:48.732 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:17:49.735 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:17:49.735 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:17:49.735 INFO:tasks.workunit.client.0.smithi172.stdout:4 2024-05-08T17:17:49.736 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-05-08T17:17:49.736 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:17:49.736 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T17:17:50.163 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:17:51.165 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:17:51.165 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:17:51.165 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-05-08T17:17:51.165 INFO:tasks.workunit.client.0.smithi172.stdout:5 2024-05-08T17:17:51.166 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:17:51.166 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T17:17:51.770 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:17:52.772 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:17:52.772 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:17:52.772 INFO:tasks.workunit.client.0.smithi172.stdout:6 2024-05-08T17:17:52.773 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-05-08T17:17:52.773 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:17:52.773 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T17:17:53.365 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:17:54.367 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:17:54.367 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:17:54.367 INFO:tasks.workunit.client.0.smithi172.stdout:7 2024-05-08T17:17:54.368 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-05-08T17:17:54.368 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:17:54.368 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T17:17:54.775 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:17:55.776 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:17:55.777 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:17:55.777 INFO:tasks.workunit.client.0.smithi172.stdout:8 2024-05-08T17:17:55.777 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-05-08T17:17:55.777 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:17:55.777 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T17:17:56.182 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:17:57.184 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:17:57.184 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:17:57.184 INFO:tasks.workunit.client.0.smithi172.stdout:9 2024-05-08T17:17:57.185 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-05-08T17:17:57.185 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:17:57.185 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T17:17:57.630 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:17:58.632 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:17:58.633 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:17:58.633 INFO:tasks.workunit.client.0.smithi172.stdout:10 2024-05-08T17:17:58.633 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-05-08T17:17:58.634 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:17:58.634 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T17:17:59.060 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:18:00.062 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:18:00.062 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:18:00.062 INFO:tasks.workunit.client.0.smithi172.stdout:11 2024-05-08T17:18:00.062 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-05-08T17:18:00.063 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:18:00.063 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T17:18:00.844 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:18:01.846 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:18:01.846 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:18:01.847 INFO:tasks.workunit.client.0.smithi172.stdout:12 2024-05-08T17:18:01.847 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-05-08T17:18:01.848 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:18:01.848 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T17:18:02.836 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:18:03.838 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:18:03.838 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:18:03.838 INFO:tasks.workunit.client.0.smithi172.stdout:13 2024-05-08T17:18:03.839 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-05-08T17:18:03.839 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:18:03.839 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T17:18:04.977 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:18:05.980 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:18:05.980 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:18:05.981 INFO:tasks.workunit.client.0.smithi172.stdout:14 2024-05-08T17:18:05.981 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-05-08T17:18:05.981 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:18:05.981 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T17:18:06.976 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:18:07.978 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:18:07.978 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:18:07.978 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-05-08T17:18:07.979 INFO:tasks.workunit.client.0.smithi172.stdout:15 2024-05-08T17:18:07.979 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:18:07.979 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T17:18:08.470 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:18:09.472 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:18:09.472 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:18:09.472 INFO:tasks.workunit.client.0.smithi172.stdout:16 2024-05-08T17:18:09.473 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-05-08T17:18:09.473 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:18:09.473 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-05-08T17:18:10.027 INFO:tasks.workunit.client.0.smithi172.stdout:osd.0 up in weight 1 up_from 6 up_thru 6 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3048841903,v1:127.0.0.1:6803/3048841903] [v2:127.0.0.1:6804/3048841903,v1:127.0.0.1:6805/3048841903] exists,up 2223a3e9-90e0-4788-a1fb-ce6b2f5035f8 2024-05-08T17:18:10.033 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-08T17:18:10.034 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-08T17:18:10.034 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-08T17:18:10.034 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-05-08T17:18:10.034 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 1 2024-05-08T17:18:10.034 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-05-08T17:18:10.034 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-05-08T17:18:10.034 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-05-08T17:18:10.034 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-05-08T17:18:10.034 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/1 2024-05-08T17:18:10.034 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=4457b0f1-028b-482b-8ee3-7ee05861e6cc --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-05-08T17:18:10.034 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-08T17:18:10.034 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-05-08T17:18:10.035 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-08T17:18:10.035 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/1' 2024-05-08T17:18:10.035 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/1/journal' 2024-05-08T17:18:10.035 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-05-08T17:18:10.035 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-05-08T17:18:10.035 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-05-08T17:18:10.035 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-05-08T17:18:10.035 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-08T17:18:10.035 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-08T17:18:10.035 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-08T17:18:10.035 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T17:18:10.035 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81110 2024-05-08T17:18:10.035 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.81110/$cluster-$name.asok' 2024-05-08T17:18:10.035 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.81110/$cluster-$name.asok' 2024-05-08T17:18:10.036 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-05-08T17:18:10.036 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-05-08T17:18:10.036 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-05-08T17:18:10.036 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-05-08T17:18:10.036 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-05-08T17:18:10.036 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-08T17:18:10.036 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-08T17:18:10.036 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-08T17:18:10.036 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-08T17:18:10.036 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-05-08T17:18:10.036 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-05-08T17:18:10.036 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/1 2024-05-08T17:18:10.036 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-05-08T17:18:10.036 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=7e5471bf-51c0-4f09-954f-75047effe049 2024-05-08T17:18:10.037 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 7e5471bf-51c0-4f09-954f-75047effe049' 2024-05-08T17:18:10.037 INFO:tasks.workunit.client.0.smithi172.stdout:add osd1 7e5471bf-51c0-4f09-954f-75047effe049 2024-05-08T17:18:10.037 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-05-08T17:18:10.052 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDSsztm/9J0AxAAP9RK7Qi3+L/GTA63hdZExQ== 2024-05-08T17:18:10.052 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDSsztm/9J0AxAAP9RK7Qi3+L/GTA63hdZExQ=="}' 2024-05-08T17:18:10.052 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 7e5471bf-51c0-4f09-954f-75047effe049 -i td/test-erasure-eio/1/new.json 2024-05-08T17:18:10.564 INFO:tasks.workunit.client.0.smithi172.stdout:1 2024-05-08T17:18:10.582 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/1/new.json 2024-05-08T17:18:10.583 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=4457b0f1-028b-482b-8ee3-7ee05861e6cc --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81110/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDSsztm/9J0AxAAP9RK7Qi3+L/GTA63hdZExQ== --osd-uuid 7e5471bf-51c0-4f09-954f-75047effe049 2024-05-08T17:18:10.610 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:18:10.614+0000 7fc6a998b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T17:18:10.612 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:18:10.618+0000 7fc6a998b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T17:18:10.613 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:18:10.618+0000 7fc6a998b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T17:18:10.613 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:18:10.618+0000 7fc6a998b7c0 -1 bluestore(td/test-erasure-eio/1/block) _read_bdev_label failed to open td/test-erasure-eio/1/block: (2) No such file or directory 2024-05-08T17:18:10.613 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:18:10.618+0000 7fc6a998b7c0 -1 bluestore(td/test-erasure-eio/1/block) _read_bdev_label failed to open td/test-erasure-eio/1/block: (2) No such file or directory 2024-05-08T17:18:10.614 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:18:10.618+0000 7fc6a998b7c0 -1 bluestore(td/test-erasure-eio/1/block) _read_bdev_label failed to open td/test-erasure-eio/1/block: (2) No such file or directory 2024-05-08T17:18:10.703 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:18:10.706+0000 7fc6a998b7c0 -1 bluestore(td/test-erasure-eio/1) _read_fsid unparsable uuid 2024-05-08T17:18:16.538 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/1/keyring 2024-05-08T17:18:16.538 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-05-08T17:18:16.540 INFO:tasks.workunit.client.0.smithi172.stdout:adding osd1 key to auth repository 2024-05-08T17:18:16.540 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-05-08T17:18:16.540 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-05-08T17:18:17.084 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-05-08T17:18:17.084 INFO:tasks.workunit.client.0.smithi172.stdout:start osd.1 2024-05-08T17:18:17.085 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=4457b0f1-028b-482b-8ee3-7ee05861e6cc --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/1 --osd-journal=td/test-erasure-eio/1/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81110/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-05-08T17:18:17.085 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-05-08T17:18:17.085 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-05-08T17:18:17.085 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-05-08T17:18:17.110 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:18:17.114+0000 7f7c162227c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T17:18:17.112 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:18:17.118+0000 7f7c162227c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T17:18:17.113 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:18:17.118+0000 7f7c162227c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T17:18:17.516 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-05-08T17:18:17.516 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-08T17:18:17.516 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-05-08T17:18:17.516 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-08T17:18:17.516 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-08T17:18:17.516 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:18:17.516 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-08T17:18:17.517 INFO:tasks.workunit.client.0.smithi172.stdout:0 2024-05-08T17:18:17.517 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:18:17.517 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T17:18:17.689 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:18:17.694+0000 7f7c162227c0 -1 Falling back to public interface 2024-05-08T17:18:17.940 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:18:18.941 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:18:18.942 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:18:18.942 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-08T17:18:18.942 INFO:tasks.workunit.client.0.smithi172.stdout:1 2024-05-08T17:18:18.942 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:18:18.942 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T17:18:19.364 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:18:20.224 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:18:20.226+0000 7f7c162227c0 -1 osd.1 0 log_to_monitors true 2024-05-08T17:18:20.365 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:18:20.366 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:18:20.366 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-08T17:18:20.366 INFO:tasks.workunit.client.0.smithi172.stdout:2 2024-05-08T17:18:20.366 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:18:20.367 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T17:18:20.907 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:18:21.908 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:18:21.908 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:18:21.908 INFO:tasks.workunit.client.0.smithi172.stdout:3 2024-05-08T17:18:21.909 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-05-08T17:18:21.909 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:18:21.909 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T17:18:22.451 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:18:23.452 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:18:23.452 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:18:23.452 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-05-08T17:18:23.453 INFO:tasks.workunit.client.0.smithi172.stdout:4 2024-05-08T17:18:23.453 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:18:23.453 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T17:18:23.987 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:18:24.989 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:18:24.990 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:18:24.990 INFO:tasks.workunit.client.0.smithi172.stdout:5 2024-05-08T17:18:24.990 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-05-08T17:18:24.991 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:18:24.991 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T17:18:25.450 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:18:26.452 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:18:26.452 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:18:26.453 INFO:tasks.workunit.client.0.smithi172.stdout:6 2024-05-08T17:18:26.453 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-05-08T17:18:26.453 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:18:26.453 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T17:18:27.178 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:18:28.179 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:18:28.179 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:18:28.180 INFO:tasks.workunit.client.0.smithi172.stdout:7 2024-05-08T17:18:28.180 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-05-08T17:18:28.180 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:18:28.180 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T17:18:28.730 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:18:29.732 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:18:29.732 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:18:29.732 INFO:tasks.workunit.client.0.smithi172.stdout:8 2024-05-08T17:18:29.733 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-05-08T17:18:29.733 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:18:29.733 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T17:18:30.152 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:18:31.153 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:18:31.154 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:18:31.154 INFO:tasks.workunit.client.0.smithi172.stdout:9 2024-05-08T17:18:31.154 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-05-08T17:18:31.154 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:18:31.155 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T17:18:31.603 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:18:32.604 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:18:32.605 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:18:32.605 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-05-08T17:18:32.605 INFO:tasks.workunit.client.0.smithi172.stdout:10 2024-05-08T17:18:32.606 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:18:32.606 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T17:18:33.043 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:18:34.044 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:18:34.045 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:18:34.045 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-05-08T17:18:34.045 INFO:tasks.workunit.client.0.smithi172.stdout:11 2024-05-08T17:18:34.045 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:18:34.045 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T17:18:34.574 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:18:35.576 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:18:35.576 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:18:35.577 INFO:tasks.workunit.client.0.smithi172.stdout:12 2024-05-08T17:18:35.577 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-05-08T17:18:35.577 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:18:35.577 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T17:18:36.275 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:18:37.276 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:18:37.276 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:18:37.277 INFO:tasks.workunit.client.0.smithi172.stdout:13 2024-05-08T17:18:37.277 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-05-08T17:18:37.277 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:18:37.277 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T17:18:38.085 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:18:39.090 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:18:39.090 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:18:39.090 INFO:tasks.workunit.client.0.smithi172.stdout:14 2024-05-08T17:18:39.091 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-05-08T17:18:39.091 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:18:39.091 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T17:18:39.799 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:18:40.801 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:18:40.801 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:18:40.802 INFO:tasks.workunit.client.0.smithi172.stdout:15 2024-05-08T17:18:40.802 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-05-08T17:18:40.802 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:18:40.802 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T17:18:41.830 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:18:42.831 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:18:42.831 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:18:42.831 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-05-08T17:18:42.832 INFO:tasks.workunit.client.0.smithi172.stdout:16 2024-05-08T17:18:42.832 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:18:42.832 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T17:18:43.639 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:18:44.641 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:18:44.641 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:18:44.642 INFO:tasks.workunit.client.0.smithi172.stdout:17 2024-05-08T17:18:44.642 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-05-08T17:18:44.642 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:18:44.642 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-05-08T17:18:45.162 INFO:tasks.workunit.client.0.smithi172.stdout:osd.1 up in weight 1 up_from 14 up_thru 14 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/444665260,v1:127.0.0.1:6811/444665260] [v2:127.0.0.1:6812/444665260,v1:127.0.0.1:6813/444665260] exists,up 7e5471bf-51c0-4f09-954f-75047effe049 2024-05-08T17:18:45.166 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-08T17:18:45.166 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-08T17:18:45.166 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-08T17:18:45.166 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-05-08T17:18:45.166 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 2 2024-05-08T17:18:45.166 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-05-08T17:18:45.166 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-05-08T17:18:45.166 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-05-08T17:18:45.166 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-05-08T17:18:45.166 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/2 2024-05-08T17:18:45.166 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=4457b0f1-028b-482b-8ee3-7ee05861e6cc --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-05-08T17:18:45.167 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-08T17:18:45.167 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-05-08T17:18:45.167 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-08T17:18:45.168 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/2' 2024-05-08T17:18:45.168 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/2/journal' 2024-05-08T17:18:45.168 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-05-08T17:18:45.168 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-05-08T17:18:45.168 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-05-08T17:18:45.168 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-05-08T17:18:45.168 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-08T17:18:45.168 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-08T17:18:45.169 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-08T17:18:45.169 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T17:18:45.169 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81110 2024-05-08T17:18:45.169 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.81110/$cluster-$name.asok' 2024-05-08T17:18:45.169 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.81110/$cluster-$name.asok' 2024-05-08T17:18:45.169 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-05-08T17:18:45.169 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-05-08T17:18:45.169 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-05-08T17:18:45.169 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-05-08T17:18:45.169 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-05-08T17:18:45.169 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-08T17:18:45.169 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-08T17:18:45.170 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-08T17:18:45.170 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-08T17:18:45.170 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-05-08T17:18:45.170 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-05-08T17:18:45.170 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/2 2024-05-08T17:18:45.170 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-05-08T17:18:45.170 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=381d00ca-355d-45de-bb71-45a02aaa8ca6 2024-05-08T17:18:45.171 INFO:tasks.workunit.client.0.smithi172.stdout:add osd2 381d00ca-355d-45de-bb71-45a02aaa8ca6 2024-05-08T17:18:45.171 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 381d00ca-355d-45de-bb71-45a02aaa8ca6' 2024-05-08T17:18:45.171 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-05-08T17:18:45.189 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQD1sztmhHSeCxAAouzEPB3PkFr3p3gjWakPUg== 2024-05-08T17:18:45.189 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQD1sztmhHSeCxAAouzEPB3PkFr3p3gjWakPUg=="}' 2024-05-08T17:18:45.189 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 381d00ca-355d-45de-bb71-45a02aaa8ca6 -i td/test-erasure-eio/2/new.json 2024-05-08T17:18:45.684 INFO:tasks.workunit.client.0.smithi172.stdout:2 2024-05-08T17:18:45.700 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/2/new.json 2024-05-08T17:18:45.702 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=4457b0f1-028b-482b-8ee3-7ee05861e6cc --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81110/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQD1sztmhHSeCxAAouzEPB3PkFr3p3gjWakPUg== --osd-uuid 381d00ca-355d-45de-bb71-45a02aaa8ca6 2024-05-08T17:18:45.728 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:18:45.729+0000 7fabfd54f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T17:18:45.744 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:18:45.745+0000 7fabfd54f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T17:18:45.745 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:18:45.749+0000 7fabfd54f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T17:18:45.746 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:18:45.749+0000 7fabfd54f7c0 -1 bluestore(td/test-erasure-eio/2/block) _read_bdev_label failed to open td/test-erasure-eio/2/block: (2) No such file or directory 2024-05-08T17:18:45.746 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:18:45.749+0000 7fabfd54f7c0 -1 bluestore(td/test-erasure-eio/2/block) _read_bdev_label failed to open td/test-erasure-eio/2/block: (2) No such file or directory 2024-05-08T17:18:45.746 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:18:45.749+0000 7fabfd54f7c0 -1 bluestore(td/test-erasure-eio/2/block) _read_bdev_label failed to open td/test-erasure-eio/2/block: (2) No such file or directory 2024-05-08T17:18:45.783 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:18:45.785+0000 7fabfd54f7c0 -1 bluestore(td/test-erasure-eio/2) _read_fsid unparsable uuid 2024-05-08T17:18:51.865 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/2/keyring 2024-05-08T17:18:51.865 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-05-08T17:18:51.866 INFO:tasks.workunit.client.0.smithi172.stdout:adding osd2 key to auth repository 2024-05-08T17:18:51.867 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-05-08T17:18:51.867 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-05-08T17:18:52.421 INFO:tasks.workunit.client.0.smithi172.stdout:start osd.2 2024-05-08T17:18:52.422 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-05-08T17:18:52.422 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=4457b0f1-028b-482b-8ee3-7ee05861e6cc --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81110/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-05-08T17:18:52.422 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-05-08T17:18:52.423 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-05-08T17:18:52.423 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-05-08T17:18:52.453 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:18:52.457+0000 7fe9191137c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T17:18:52.490 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:18:52.493+0000 7fe9191137c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T17:18:52.491 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:18:52.493+0000 7fe9191137c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T17:18:52.848 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-05-08T17:18:52.849 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-08T17:18:52.849 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-05-08T17:18:52.849 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-08T17:18:52.849 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-08T17:18:52.849 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:18:52.849 INFO:tasks.workunit.client.0.smithi172.stdout:0 2024-05-08T17:18:52.850 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-08T17:18:52.850 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:18:52.850 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T17:18:53.046 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:18:53.049+0000 7fe9191137c0 -1 Falling back to public interface 2024-05-08T17:18:53.468 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:18:54.470 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:18:54.471 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:18:54.471 INFO:tasks.workunit.client.0.smithi172.stdout:1 2024-05-08T17:18:54.471 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-08T17:18:54.472 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:18:54.472 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T17:18:54.880 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:18:55.539 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:18:55.541+0000 7fe9191137c0 -1 osd.2 0 log_to_monitors true 2024-05-08T17:18:55.882 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:18:55.882 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:18:55.882 INFO:tasks.workunit.client.0.smithi172.stdout:2 2024-05-08T17:18:55.883 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-08T17:18:55.883 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:18:55.883 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T17:18:56.372 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:18:57.373 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:18:57.374 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:18:57.374 INFO:tasks.workunit.client.0.smithi172.stdout:3 2024-05-08T17:18:57.374 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-05-08T17:18:57.374 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:18:57.375 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T17:18:57.846 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:18:58.848 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:18:58.848 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:18:58.848 INFO:tasks.workunit.client.0.smithi172.stdout:4 2024-05-08T17:18:58.849 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-05-08T17:18:58.849 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:18:58.849 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T17:18:59.327 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:19:00.329 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:19:00.329 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:19:00.330 INFO:tasks.workunit.client.0.smithi172.stdout:5 2024-05-08T17:19:00.330 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-05-08T17:19:00.330 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:19:00.330 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T17:19:00.805 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:19:01.806 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:19:01.806 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:19:01.807 INFO:tasks.workunit.client.0.smithi172.stdout:6 2024-05-08T17:19:01.807 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-05-08T17:19:01.807 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:19:01.807 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T17:19:02.350 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:19:03.351 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:19:03.352 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:19:03.352 INFO:tasks.workunit.client.0.smithi172.stdout:7 2024-05-08T17:19:03.352 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-05-08T17:19:03.352 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:19:03.352 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T17:19:03.790 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:19:04.792 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:19:04.792 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:19:04.792 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-05-08T17:19:04.792 INFO:tasks.workunit.client.0.smithi172.stdout:8 2024-05-08T17:19:04.793 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:19:04.793 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T17:19:05.400 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:19:06.402 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:19:06.402 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:19:06.403 INFO:tasks.workunit.client.0.smithi172.stdout:9 2024-05-08T17:19:06.403 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-05-08T17:19:06.403 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:19:06.403 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T17:19:06.857 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:19:07.859 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:19:07.859 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:19:07.860 INFO:tasks.workunit.client.0.smithi172.stdout:10 2024-05-08T17:19:07.861 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-05-08T17:19:07.861 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:19:07.861 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T17:19:08.871 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:19:09.873 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:19:09.873 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:19:09.873 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-05-08T17:19:09.873 INFO:tasks.workunit.client.0.smithi172.stdout:11 2024-05-08T17:19:09.874 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:19:09.874 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T17:19:10.722 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:19:11.724 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:19:11.724 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:19:11.724 INFO:tasks.workunit.client.0.smithi172.stdout:12 2024-05-08T17:19:11.725 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-05-08T17:19:11.725 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:19:11.725 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T17:19:12.552 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:19:13.554 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:19:13.554 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:19:13.554 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-05-08T17:19:13.554 INFO:tasks.workunit.client.0.smithi172.stdout:13 2024-05-08T17:19:13.555 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:19:13.555 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T17:19:14.332 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:19:15.333 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:19:15.334 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:19:15.334 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-05-08T17:19:15.334 INFO:tasks.workunit.client.0.smithi172.stdout:14 2024-05-08T17:19:15.334 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:19:15.334 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T17:19:16.349 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:19:17.351 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:19:17.351 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:19:17.351 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-05-08T17:19:17.351 INFO:tasks.workunit.client.0.smithi172.stdout:15 2024-05-08T17:19:17.352 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:19:17.352 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T17:19:18.199 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:19:19.201 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:19:19.202 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:19:19.202 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-05-08T17:19:19.202 INFO:tasks.workunit.client.0.smithi172.stdout:16 2024-05-08T17:19:19.203 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:19:19.203 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T17:19:19.757 INFO:tasks.workunit.client.0.smithi172.stdout:osd.2 up in weight 1 up_from 19 up_thru 19 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/3175495318,v1:127.0.0.1:6819/3175495318] [v2:127.0.0.1:6820/3175495318,v1:127.0.0.1:6821/3175495318] exists,up 381d00ca-355d-45de-bb71-45a02aaa8ca6 2024-05-08T17:19:19.760 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-08T17:19:19.760 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-08T17:19:19.760 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-08T17:19:19.760 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:50: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-05-08T17:19:19.760 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:51: setup_osds: run_osd td/test-erasure-eio 3 2024-05-08T17:19:19.760 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-erasure-eio 2024-05-08T17:19:19.760 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-05-08T17:19:19.760 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2024-05-08T17:19:19.760 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-05-08T17:19:19.760 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-erasure-eio/3 2024-05-08T17:19:19.760 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=4457b0f1-028b-482b-8ee3-7ee05861e6cc --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-05-08T17:19:19.761 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-08T17:19:19.761 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-05-08T17:19:19.761 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-08T17:19:19.761 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-erasure-eio/3' 2024-05-08T17:19:19.761 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/3/journal' 2024-05-08T17:19:19.761 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-05-08T17:19:19.761 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-05-08T17:19:19.761 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-05-08T17:19:19.761 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-05-08T17:19:19.761 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-08T17:19:19.761 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-08T17:19:19.761 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-08T17:19:19.761 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T17:19:19.761 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81110 2024-05-08T17:19:19.761 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.81110/$cluster-$name.asok' 2024-05-08T17:19:19.762 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.81110/$cluster-$name.asok' 2024-05-08T17:19:19.762 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-05-08T17:19:19.763 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-05-08T17:19:19.763 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-05-08T17:19:19.763 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-05-08T17:19:19.763 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-05-08T17:19:19.763 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-08T17:19:19.763 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-08T17:19:19.763 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-08T17:19:19.763 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-08T17:19:19.763 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-05-08T17:19:19.763 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-05-08T17:19:19.763 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-erasure-eio/3 2024-05-08T17:19:19.763 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-05-08T17:19:19.764 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=a98f661e-5b93-46fd-8f6c-20308c8578f1 2024-05-08T17:19:19.764 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 a98f661e-5b93-46fd-8f6c-20308c8578f1' 2024-05-08T17:19:19.764 INFO:tasks.workunit.client.0.smithi172.stdout:add osd3 a98f661e-5b93-46fd-8f6c-20308c8578f1 2024-05-08T17:19:19.764 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-05-08T17:19:19.786 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAXtDtmqFUwLxAAr05iWcy9MG94wyQxka/s+Q== 2024-05-08T17:19:19.786 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAXtDtmqFUwLxAAr05iWcy9MG94wyQxka/s+Q=="}' 2024-05-08T17:19:19.786 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new a98f661e-5b93-46fd-8f6c-20308c8578f1 -i td/test-erasure-eio/3/new.json 2024-05-08T17:19:20.369 INFO:tasks.workunit.client.0.smithi172.stdout:3 2024-05-08T17:19:20.385 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-erasure-eio/3/new.json 2024-05-08T17:19:20.386 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=4457b0f1-028b-482b-8ee3-7ee05861e6cc --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81110/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAXtDtmqFUwLxAAr05iWcy9MG94wyQxka/s+Q== --osd-uuid a98f661e-5b93-46fd-8f6c-20308c8578f1 2024-05-08T17:19:20.420 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:19:20.420+0000 7f05976a47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T17:19:20.438 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:19:20.440+0000 7f05976a47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T17:19:20.439 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:19:20.440+0000 7f05976a47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T17:19:20.440 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:19:20.440+0000 7f05976a47c0 -1 bluestore(td/test-erasure-eio/3/block) _read_bdev_label failed to open td/test-erasure-eio/3/block: (2) No such file or directory 2024-05-08T17:19:20.440 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:19:20.440+0000 7f05976a47c0 -1 bluestore(td/test-erasure-eio/3/block) _read_bdev_label failed to open td/test-erasure-eio/3/block: (2) No such file or directory 2024-05-08T17:19:20.440 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:19:20.440+0000 7f05976a47c0 -1 bluestore(td/test-erasure-eio/3/block) _read_bdev_label failed to open td/test-erasure-eio/3/block: (2) No such file or directory 2024-05-08T17:19:20.546 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:19:20.548+0000 7f05976a47c0 -1 bluestore(td/test-erasure-eio/3) _read_fsid unparsable uuid 2024-05-08T17:19:26.597 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-erasure-eio/3/keyring 2024-05-08T17:19:26.598 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-05-08T17:19:26.599 INFO:tasks.workunit.client.0.smithi172.stdout:adding osd3 key to auth repository 2024-05-08T17:19:26.599 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2024-05-08T17:19:26.599 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-erasure-eio/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-05-08T17:19:27.148 INFO:tasks.workunit.client.0.smithi172.stdout:start osd.3 2024-05-08T17:19:27.149 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2024-05-08T17:19:27.149 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-05-08T17:19:27.149 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-05-08T17:19:27.149 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=4457b0f1-028b-482b-8ee3-7ee05861e6cc --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/3 --osd-journal=td/test-erasure-eio/3/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81110/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-05-08T17:19:27.149 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-05-08T17:19:27.183 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:19:27.184+0000 7fe376a587c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T17:19:27.227 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:19:27.228+0000 7fe376a587c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T17:19:27.228 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:19:27.232+0000 7fe376a587c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T17:19:27.529 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2024-05-08T17:19:27.529 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-08T17:19:27.529 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-05-08T17:19:27.529 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-08T17:19:27.529 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-08T17:19:27.530 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:19:27.530 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-08T17:19:27.530 INFO:tasks.workunit.client.0.smithi172.stdout:0 2024-05-08T17:19:27.530 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:19:27.531 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T17:19:27.779 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:19:27.780+0000 7fe376a587c0 -1 Falling back to public interface 2024-05-08T17:19:27.962 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:19:28.964 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:19:28.964 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:19:28.964 INFO:tasks.workunit.client.0.smithi172.stdout:1 2024-05-08T17:19:28.965 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-08T17:19:28.965 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:19:28.965 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T17:19:29.382 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:19:30.336 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:19:30.336+0000 7fe376a587c0 -1 osd.3 0 log_to_monitors true 2024-05-08T17:19:30.384 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:19:30.385 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:19:30.385 INFO:tasks.workunit.client.0.smithi172.stdout:2 2024-05-08T17:19:30.385 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-08T17:19:30.386 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:19:30.386 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T17:19:31.052 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:19:32.054 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:19:32.054 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:19:32.054 INFO:tasks.workunit.client.0.smithi172.stdout:3 2024-05-08T17:19:32.055 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-05-08T17:19:32.055 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:19:32.055 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T17:19:32.554 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:19:33.556 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:19:33.557 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:19:33.557 INFO:tasks.workunit.client.0.smithi172.stdout:4 2024-05-08T17:19:33.557 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-05-08T17:19:33.558 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:19:33.558 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T17:19:33.964 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:19:34.965 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:19:34.965 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:19:34.966 INFO:tasks.workunit.client.0.smithi172.stdout:5 2024-05-08T17:19:34.966 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-05-08T17:19:34.966 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:19:34.966 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T17:19:35.433 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:19:36.436 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:19:36.436 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:19:36.436 INFO:tasks.workunit.client.0.smithi172.stdout:6 2024-05-08T17:19:36.436 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-05-08T17:19:36.437 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:19:36.437 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T17:19:36.882 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:19:37.883 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:19:37.884 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:19:37.884 INFO:tasks.workunit.client.0.smithi172.stdout:7 2024-05-08T17:19:37.884 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-05-08T17:19:37.885 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:19:37.885 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T17:19:38.363 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:19:39.365 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:19:39.365 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:19:39.366 INFO:tasks.workunit.client.0.smithi172.stdout:8 2024-05-08T17:19:39.366 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-05-08T17:19:39.366 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:19:39.366 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T17:19:39.777 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:19:40.779 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:19:40.779 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:19:40.779 INFO:tasks.workunit.client.0.smithi172.stdout:9 2024-05-08T17:19:40.780 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-05-08T17:19:40.780 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:19:40.780 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T17:19:41.248 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:19:42.249 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:19:42.250 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:19:42.250 INFO:tasks.workunit.client.0.smithi172.stdout:10 2024-05-08T17:19:42.250 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-05-08T17:19:42.251 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:19:42.251 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T17:19:42.796 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:19:43.798 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:19:43.802 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:19:43.802 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-05-08T17:19:43.802 INFO:tasks.workunit.client.0.smithi172.stdout:11 2024-05-08T17:19:43.803 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:19:43.803 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T17:19:44.836 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:19:45.838 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:19:45.838 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:19:45.838 INFO:tasks.workunit.client.0.smithi172.stdout:12 2024-05-08T17:19:45.839 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-05-08T17:19:45.839 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:19:45.839 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T17:19:46.537 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:19:47.538 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:19:47.538 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:19:47.539 INFO:tasks.workunit.client.0.smithi172.stdout:13 2024-05-08T17:19:47.539 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-05-08T17:19:47.539 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:19:47.539 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T17:19:48.231 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:19:49.233 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:19:49.233 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:19:49.233 INFO:tasks.workunit.client.0.smithi172.stdout:14 2024-05-08T17:19:49.234 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-05-08T17:19:49.234 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:19:49.234 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T17:19:50.230 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:19:51.239 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:19:51.239 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:19:51.239 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-05-08T17:19:51.239 INFO:tasks.workunit.client.0.smithi172.stdout:15 2024-05-08T17:19:51.240 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:19:51.240 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T17:19:52.036 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:19:53.038 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:19:53.038 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:19:53.038 INFO:tasks.workunit.client.0.smithi172.stdout:16 2024-05-08T17:19:53.039 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-05-08T17:19:53.039 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:19:53.039 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T17:19:53.811 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:19:54.812 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:19:54.812 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:19:54.813 INFO:tasks.workunit.client.0.smithi172.stdout:17 2024-05-08T17:19:54.813 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-05-08T17:19:54.813 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:19:54.813 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-05-08T17:19:55.352 INFO:tasks.workunit.client.0.smithi172.stdout:osd.3 up in weight 1 up_from 28 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/2305361935,v1:127.0.0.1:6827/2305361935] [v2:127.0.0.1:6828/2305361935,v1:127.0.0.1:6829/2305361935] exists,up a98f661e-5b93-46fd-8f6c-20308c8578f1 2024-05-08T17:19:55.353 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-08T17:19:55.353 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-08T17:19:55.353 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-08T17:19:55.354 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: get_asok_path osd.0 2024-05-08T17:19:55.354 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-05-08T17:19:55.354 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-05-08T17:19:55.354 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-08T17:19:55.354 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T17:19:55.354 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81110 2024-05-08T17:19:55.354 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.81110/ceph-osd.0.asok 2024-05-08T17:19:55.355 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: CEPH_ARGS= 2024-05-08T17:19:55.355 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:55: setup_osds: ceph --admin-daemon /tmp/ceph-asok.81110/ceph-osd.0.asok log flush 2024-05-08T17:19:55.453 INFO:tasks.workunit.client.0.smithi172.stdout:{} 2024-05-08T17:19:55.465 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:56: setup_osds: grep 'load: jerasure.*lrc' td/test-erasure-eio/osd.0.log 2024-05-08T17:19:55.467 INFO:tasks.workunit.client.0.smithi172.stdout:2024-05-08T17:17:44.091+0000 7f419bb7c7c0 0 load: jerasure load: lrc 2024-05-08T17:19:55.473 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:349: TEST_rados_get_with_subreadall_eio_shard_1: local poolname=pool-jerasure 2024-05-08T17:19:55.473 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:350: TEST_rados_get_with_subreadall_eio_shard_1: create_erasure_coded_pool pool-jerasure 2 1 2024-05-08T17:19:55.473 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:67: create_erasure_coded_pool: local poolname=pool-jerasure 2024-05-08T17:19:55.473 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:68: create_erasure_coded_pool: shift 2024-05-08T17:19:55.473 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:69: create_erasure_coded_pool: local k=2 2024-05-08T17:19:55.473 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:70: create_erasure_coded_pool: shift 2024-05-08T17:19:55.473 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:71: create_erasure_coded_pool: local m=1 2024-05-08T17:19:55.473 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:72: create_erasure_coded_pool: shift 2024-05-08T17:19:55.473 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:74: create_erasure_coded_pool: ceph osd erasure-code-profile set myprofile plugin=jerasure k=2 m=1 crush-failure-domain=osd 2024-05-08T17:19:56.676 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:78: create_erasure_coded_pool: create_pool pool-jerasure 1 1 erasure myprofile 2024-05-08T17:19:56.676 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create pool-jerasure 1 1 erasure myprofile 2024-05-08T17:19:57.759 INFO:tasks.workunit.client.0.smithi172.stderr:pool 'pool-jerasure' already exists 2024-05-08T17:19:57.777 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-05-08T17:19:58.780 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:80: create_erasure_coded_pool: wait_for_clean 2024-05-08T17:19:58.780 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-05-08T17:19:58.780 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-05-08T17:19:58.780 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-05-08T17:19:58.780 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-05-08T17:19:58.781 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-05-08T17:19:58.781 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-05-08T17:19:58.782 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-05-08T17:19:58.782 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-05-08T17:19:58.782 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-05-08T17:19:58.910 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-05-08T17:19:58.910 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-05-08T17:19:58.910 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-05-08T17:19:58.910 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-05-08T17:19:58.910 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-05-08T17:19:58.911 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-05-08T17:19:59.357 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-05-08T17:19:59.357 INFO:tasks.workunit.client.0.smithi172.stderr:1 2024-05-08T17:19:59.357 INFO:tasks.workunit.client.0.smithi172.stderr:2 2024-05-08T17:19:59.357 INFO:tasks.workunit.client.0.smithi172.stderr:3' 2024-05-08T17:19:59.357 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-05-08T17:19:59.357 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T17:19:59.358 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-05-08T17:19:59.519 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=25769803807 2024-05-08T17:19:59.519 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 25769803807 2024-05-08T17:19:59.519 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803807' 2024-05-08T17:19:59.519 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T17:19:59.519 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-05-08T17:19:59.671 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=60129542162 2024-05-08T17:19:59.671 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 60129542162 2024-05-08T17:19:59.672 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803807 1-60129542162' 2024-05-08T17:19:59.672 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T17:19:59.672 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-05-08T17:19:59.828 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=81604378639 2024-05-08T17:19:59.828 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 81604378639 2024-05-08T17:19:59.828 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803807 1-60129542162 2-81604378639' 2024-05-08T17:19:59.828 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T17:19:59.828 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-05-08T17:19:59.988 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=120259084293 2024-05-08T17:19:59.988 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 120259084293 2024-05-08T17:19:59.988 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803807 1-60129542162 2-81604378639 3-120259084293' 2024-05-08T17:19:59.988 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-08T17:19:59.989 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-25769803807 2024-05-08T17:19:59.989 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-08T17:19:59.991 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-05-08T17:19:59.992 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-25769803807 2024-05-08T17:19:59.992 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-08T17:19:59.993 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=25769803807 2024-05-08T17:19:59.993 INFO:tasks.workunit.client.0.smithi172.stdout:waiting osd.0 seq 25769803807 2024-05-08T17:19:59.994 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 25769803807' 2024-05-08T17:19:59.994 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-05-08T17:20:00.399 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 25769803807 -lt 25769803807 2024-05-08T17:20:00.399 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-08T17:20:00.400 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-60129542162 2024-05-08T17:20:00.400 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-08T17:20:00.402 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-05-08T17:20:00.402 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-60129542162 2024-05-08T17:20:00.403 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-08T17:20:00.404 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=60129542162 2024-05-08T17:20:00.404 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 60129542162' 2024-05-08T17:20:00.404 INFO:tasks.workunit.client.0.smithi172.stdout:waiting osd.1 seq 60129542162 2024-05-08T17:20:00.404 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-05-08T17:20:00.853 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 60129542163 -lt 60129542162 2024-05-08T17:20:00.853 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-08T17:20:00.854 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-81604378639 2024-05-08T17:20:00.854 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-08T17:20:00.856 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-05-08T17:20:00.857 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-81604378639 2024-05-08T17:20:00.857 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-08T17:20:00.858 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=81604378639 2024-05-08T17:20:00.858 INFO:tasks.workunit.client.0.smithi172.stdout:waiting osd.2 seq 81604378639 2024-05-08T17:20:00.859 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 81604378639' 2024-05-08T17:20:00.859 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-05-08T17:20:01.285 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 81604378639 -lt 81604378639 2024-05-08T17:20:01.285 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-08T17:20:01.286 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-120259084293 2024-05-08T17:20:01.286 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-08T17:20:01.288 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-05-08T17:20:01.289 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-120259084293 2024-05-08T17:20:01.289 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-08T17:20:01.290 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=120259084293 2024-05-08T17:20:01.291 INFO:tasks.workunit.client.0.smithi172.stdout:waiting osd.3 seq 120259084293 2024-05-08T17:20:01.291 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 120259084293' 2024-05-08T17:20:01.291 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-05-08T17:20:01.726 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 120259084291 -lt 120259084293 2024-05-08T17:20:01.727 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-05-08T17:20:02.728 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-05-08T17:20:02.729 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-05-08T17:20:03.138 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 120259084293 -lt 120259084293 2024-05-08T17:20:03.139 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-05-08T17:20:03.139 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-05-08T17:20:03.140 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-05-08T17:20:03.696 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-05-08T17:20:03.697 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-05-08T17:20:03.697 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-05-08T17:20:03.697 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-05-08T17:20:03.697 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-05-08T17:20:03.697 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-05-08T17:20:03.697 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-05-08T17:20:03.698 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-05-08T17:20:04.118 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-05-08T17:20:04.119 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-05-08T17:20:04.119 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-05-08T17:20:04.119 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-05-08T17:20:04.694 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-05-08T17:20:04.694 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-05-08T17:20:04.694 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-05-08T17:20:04.694 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:352: TEST_rados_get_with_subreadall_eio_shard_1: rados_put_get_data eio td/test-erasure-eio 1 recovery 2024-05-08T17:20:04.695 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:147: rados_put_get_data: local inject=eio 2024-05-08T17:20:04.695 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:148: rados_put_get_data: shift 2024-05-08T17:20:04.695 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:149: rados_put_get_data: local dir=td/test-erasure-eio 2024-05-08T17:20:04.695 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:150: rados_put_get_data: shift 2024-05-08T17:20:04.695 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:151: rados_put_get_data: local shard_id=1 2024-05-08T17:20:04.695 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:152: rados_put_get_data: shift 2024-05-08T17:20:04.695 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:153: rados_put_get_data: local arg=recovery 2024-05-08T17:20:04.695 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:157: rados_put_get_data: local poolname=pool-jerasure 2024-05-08T17:20:04.695 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:158: rados_put_get_data: local objname=obj-eio-81110-1 2024-05-08T17:20:04.695 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:159: rados_put_get_data: rados_put td/test-erasure-eio pool-jerasure obj-eio-81110-1 2024-05-08T17:20:04.695 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:90: rados_put: local dir=td/test-erasure-eio 2024-05-08T17:20:04.695 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:91: rados_put: local poolname=pool-jerasure 2024-05-08T17:20:04.695 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:92: rados_put: local objname=obj-eio-81110-1 2024-05-08T17:20:04.695 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-05-08T17:20:04.695 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 AAA 2024-05-08T17:20:04.696 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-05-08T17:20:04.696 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 BBB 2024-05-08T17:20:04.696 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-05-08T17:20:04.696 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 CCCC 2024-05-08T17:20:04.696 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:94: rados_put: for marker in AAA BBB CCCC DDDD 2024-05-08T17:20:04.696 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:95: rados_put: printf '%*s' 1024 DDDD 2024-05-08T17:20:04.696 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:100: rados_put: rados --pool pool-jerasure put obj-eio-81110-1 td/test-erasure-eio/ORIGINAL 2024-05-08T17:20:04.842 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:160: rados_put_get_data: inject_eio ec data pool-jerasure obj-eio-81110-1 td/test-erasure-eio 1 2024-05-08T17:20:04.843 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2394: inject_eio: local pooltype=ec 2024-05-08T17:20:04.843 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2395: inject_eio: shift 2024-05-08T17:20:04.843 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2396: inject_eio: local which=data 2024-05-08T17:20:04.843 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2397: inject_eio: shift 2024-05-08T17:20:04.843 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2398: inject_eio: local poolname=pool-jerasure 2024-05-08T17:20:04.843 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2399: inject_eio: shift 2024-05-08T17:20:04.843 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2400: inject_eio: local objname=obj-eio-81110-1 2024-05-08T17:20:04.843 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2401: inject_eio: shift 2024-05-08T17:20:04.843 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2402: inject_eio: local dir=td/test-erasure-eio 2024-05-08T17:20:04.843 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2403: inject_eio: shift 2024-05-08T17:20:04.843 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2404: inject_eio: local shard_id=1 2024-05-08T17:20:04.843 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2405: inject_eio: shift 2024-05-08T17:20:04.843 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: get_osds pool-jerasure obj-eio-81110-1 2024-05-08T17:20:04.844 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2024-05-08T17:20:04.844 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-eio-81110-1 2024-05-08T17:20:04.844 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-05-08T17:20:04.844 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure obj-eio-81110-1 2024-05-08T17:20:05.243 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-05-08T17:20:05.244 INFO:tasks.workunit.client.0.smithi172.stderr:1 2024-05-08T17:20:05.244 INFO:tasks.workunit.client.0.smithi172.stderr:2' 2024-05-08T17:20:05.244 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 2 2024-05-08T17:20:05.244 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: initial_osds=('3' '1' '2') 2024-05-08T17:20:05.244 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: local -a initial_osds 2024-05-08T17:20:05.244 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2408: inject_eio: local osd_id=1 2024-05-08T17:20:05.244 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2409: inject_eio: '[' ec '!=' ec ']' 2024-05-08T17:20:05.244 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: cat td/test-erasure-eio/1/type 2024-05-08T17:20:05.246 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: type=bluestore 2024-05-08T17:20:05.246 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2024-05-08T17:20:05.246 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-05-08T17:20:05.246 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=1 2024-05-08T17:20:05.246 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-05-08T17:20:05.246 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-05-08T17:20:05.246 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-05-08T17:20:05.247 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.1 2024-05-08T17:20:05.247 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-05-08T17:20:05.247 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-05-08T17:20:05.248 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-08T17:20:05.248 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T17:20:05.248 INFO:tasks.workunit.client.0.smithi172.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81110 2024-05-08T17:20:05.248 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.81110/ceph-osd.1.asok 2024-05-08T17:20:05.248 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.81110/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2024-05-08T17:20:05.371 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-05-08T17:20:05.371 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local loop=0 2024-05-08T17:20:05.373 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: get_asok_path osd.1 2024-05-08T17:20:05.373 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: grep -q Invalid 2024-05-08T17:20:05.373 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-05-08T17:20:05.373 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-05-08T17:20:05.373 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-05-08T17:20:05.373 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T17:20:05.373 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81110 2024-05-08T17:20:05.373 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.81110/ceph-osd.1.asok 2024-05-08T17:20:05.374 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: CEPH_ARGS= 2024-05-08T17:20:05.374 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: ceph --admin-daemon /tmp/ceph-asok.81110/ceph-osd.1.asok injectdataerr pool-jerasure obj-eio-81110-1 1 2024-05-08T17:20:05.493 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:161: rados_put_get_data: rados_get td/test-erasure-eio pool-jerasure obj-eio-81110-1 2024-05-08T17:20:05.493 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:104: rados_get: local dir=td/test-erasure-eio 2024-05-08T17:20:05.493 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:105: rados_get: local poolname=pool-jerasure 2024-05-08T17:20:05.493 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:106: rados_get: local objname=obj-eio-81110-1 2024-05-08T17:20:05.493 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:107: rados_get: local expect=ok 2024-05-08T17:20:05.494 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:112: rados_get: '[' ok = fail ']' 2024-05-08T17:20:05.494 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:120: rados_get: rados --pool pool-jerasure get obj-eio-81110-1 td/test-erasure-eio/COPY 2024-05-08T17:20:05.530 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:121: rados_get: diff td/test-erasure-eio/ORIGINAL td/test-erasure-eio/COPY 2024-05-08T17:20:05.530 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:122: rados_get: rm td/test-erasure-eio/COPY 2024-05-08T17:20:05.531 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:163: rados_put_get_data: '[' recovery = recovery ']' 2024-05-08T17:20:05.532 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:170: rados_put_get_data: get_osds pool-jerasure obj-eio-81110-1 2024-05-08T17:20:05.532 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2024-05-08T17:20:05.532 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-eio-81110-1 2024-05-08T17:20:05.532 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure obj-eio-81110-1 2024-05-08T17:20:05.533 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-05-08T17:20:05.943 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-05-08T17:20:05.943 INFO:tasks.workunit.client.0.smithi172.stderr:1 2024-05-08T17:20:05.943 INFO:tasks.workunit.client.0.smithi172.stderr:2' 2024-05-08T17:20:05.943 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 2 2024-05-08T17:20:05.943 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:170: rados_put_get_data: initial_osds=('3' '1' '2') 2024-05-08T17:20:05.943 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:170: rados_put_get_data: local -a initial_osds 2024-05-08T17:20:05.944 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:171: rados_put_get_data: local last_osd=2 2024-05-08T17:20:05.944 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:173: rados_put_get_data: kill_daemons td/test-erasure-eio TERM osd.2 2024-05-08T17:20:05.944 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-08T17:20:05.944 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-08T17:20:05.944 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-08T17:20:05.944 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-08T17:20:05.945 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-08T17:20:06.252 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-08T17:20:06.253 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:174: rados_put_get_data: ceph osd out 2 2024-05-08T17:20:07.352 INFO:tasks.workunit.client.0.smithi172.stderr:osd.2 is already out. 2024-05-08T17:20:07.370 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:175: rados_put_get_data: get_osds pool-jerasure obj-eio-81110-1 2024-05-08T17:20:07.370 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:175: rados_put_get_data: grep '\<2\>' 2024-05-08T17:20:07.370 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-jerasure 2024-05-08T17:20:07.371 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-eio-81110-1 2024-05-08T17:20:07.371 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-jerasure obj-eio-81110-1 2024-05-08T17:20:07.371 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-05-08T17:20:07.936 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-05-08T17:20:07.936 INFO:tasks.workunit.client.0.smithi172.stderr:1 2024-05-08T17:20:07.936 INFO:tasks.workunit.client.0.smithi172.stderr:0' 2024-05-08T17:20:07.937 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-05-08T17:20:07.937 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:176: rados_put_get_data: ceph osd in 2 2024-05-08T17:20:09.037 INFO:tasks.workunit.client.0.smithi172.stderr:osd.2 is already in. 2024-05-08T17:20:09.055 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:177: rados_put_get_data: activate_osd td/test-erasure-eio 2 2024-05-08T17:20:09.056 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/test-erasure-eio 2024-05-08T17:20:09.056 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-05-08T17:20:09.056 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-05-08T17:20:09.056 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-05-08T17:20:09.056 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/test-erasure-eio/2 2024-05-08T17:20:09.056 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=4457b0f1-028b-482b-8ee3-7ee05861e6cc --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true ' 2024-05-08T17:20:09.056 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-05-08T17:20:09.056 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-05-08T17:20:09.056 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-05-08T17:20:09.056 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/test-erasure-eio/2' 2024-05-08T17:20:09.056 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/test-erasure-eio/2/journal' 2024-05-08T17:20:09.056 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-05-08T17:20:09.056 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-05-08T17:20:09.057 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/test-erasure-eio' 2024-05-08T17:20:09.058 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-05-08T17:20:09.058 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-05-08T17:20:09.058 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-05-08T17:20:09.058 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-05-08T17:20:09.059 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T17:20:09.059 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81110 2024-05-08T17:20:09.059 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.81110/$cluster-$name.asok' 2024-05-08T17:20:09.059 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.81110/$cluster-$name.asok' 2024-05-08T17:20:09.059 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-05-08T17:20:09.059 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/test-erasure-eio/$name.log' 2024-05-08T17:20:09.059 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/test-erasure-eio/$name.pid' 2024-05-08T17:20:09.059 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-05-08T17:20:09.059 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-05-08T17:20:09.059 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-05-08T17:20:09.059 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-05-08T17:20:09.059 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-05-08T17:20:09.059 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-05-08T17:20:09.059 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/test-erasure-eio/2 2024-05-08T17:20:09.060 INFO:tasks.workunit.client.0.smithi172.stdout:start osd.2 2024-05-08T17:20:09.061 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-05-08T17:20:09.061 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=4457b0f1-028b-482b-8ee3-7ee05861e6cc --auth-supported=none --mon-host=127.0.0.1:7112 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-erasure-eio/2 --osd-journal=td/test-erasure-eio/2/journal --chdir= --run-dir=td/test-erasure-eio '--admin-socket=/tmp/ceph-asok.81110/$cluster-$name.asok' --debug-osd=20 '--log-file=td/test-erasure-eio/$name.log' '--pid-file=td/test-erasure-eio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-05-08T17:20:09.061 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/test-erasure-eio/2/whoami 2024-05-08T17:20:09.061 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-05-08T17:20:09.061 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-05-08T17:20:09.061 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-05-08T17:20:09.061 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-05-08T17:20:09.090 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:20:09.095+0000 7fe769b2d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T17:20:09.271 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:20:09.275+0000 7fe769b2d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T17:20:09.272 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:20:09.275+0000 7fe769b2d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-05-08T17:20:09.626 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-05-08T17:20:09.626 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-05-08T17:20:09.627 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-05-08T17:20:09.627 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-05-08T17:20:09.627 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-05-08T17:20:09.627 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:20:09.627 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-05-08T17:20:09.628 INFO:tasks.workunit.client.0.smithi172.stdout:0 2024-05-08T17:20:09.628 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:20:09.629 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T17:20:09.818 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:20:09.823+0000 7fe769b2d7c0 -1 Falling back to public interface 2024-05-08T17:20:10.143 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:20:11.145 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:20:11.146 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:20:11.146 INFO:tasks.workunit.client.0.smithi172.stdout:1 2024-05-08T17:20:11.146 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-05-08T17:20:11.147 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:20:11.147 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T17:20:11.570 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:20:12.572 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:20:12.572 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:20:12.572 INFO:tasks.workunit.client.0.smithi172.stdout:2 2024-05-08T17:20:12.573 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-05-08T17:20:12.573 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:20:12.573 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T17:20:12.860 INFO:tasks.workunit.client.0.smithi172.stderr:2024-05-08T17:20:12.863+0000 7fe769b2d7c0 -1 osd.2 37 log_to_monitors true 2024-05-08T17:20:12.977 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-05-08T17:20:13.978 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-05-08T17:20:13.978 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-05-08T17:20:13.979 INFO:tasks.workunit.client.0.smithi172.stdout:3 2024-05-08T17:20:13.979 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-05-08T17:20:13.979 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-05-08T17:20:13.979 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-05-08T17:20:14.633 INFO:tasks.workunit.client.0.smithi172.stdout:osd.2 up in weight 1 up_from 50 up_thru 19 down_at 38 last_clean_interval [19,37) [v2:127.0.0.1:6818/1791421535,v1:127.0.0.1:6819/1791421535] [v2:127.0.0.1:6820/1791421535,v1:127.0.0.1:6821/1791421535] exists,up 381d00ca-355d-45de-bb71-45a02aaa8ca6 2024-05-08T17:20:14.635 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-05-08T17:20:14.635 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-05-08T17:20:14.635 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-05-08T17:20:14.635 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:178: rados_put_get_data: wait_for_clean 2024-05-08T17:20:14.636 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-05-08T17:20:14.636 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-05-08T17:20:14.636 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-05-08T17:20:14.636 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-05-08T17:20:14.636 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-05-08T17:20:14.636 INFO:tasks.workunit.client.0.smithi172.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-05-08T17:20:14.636 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-05-08T17:20:14.636 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-05-08T17:20:14.636 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-05-08T17:20:14.754 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-05-08T17:20:14.754 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-05-08T17:20:14.754 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-05-08T17:20:14.754 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-05-08T17:20:14.754 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2197: flush_pg_stats: local timeout=300 2024-05-08T17:20:14.754 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ceph osd ls 2024-05-08T17:20:15.420 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2199: flush_pg_stats: ids='0 2024-05-08T17:20:15.420 INFO:tasks.workunit.client.0.smithi172.stderr:1 2024-05-08T17:20:15.420 INFO:tasks.workunit.client.0.smithi172.stderr:2 2024-05-08T17:20:15.420 INFO:tasks.workunit.client.0.smithi172.stderr:3' 2024-05-08T17:20:15.420 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2200: flush_pg_stats: seqs= 2024-05-08T17:20:15.420 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T17:20:15.421 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-05-08T17:20:15.642 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=25769803812 2024-05-08T17:20:15.643 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 25769803812 2024-05-08T17:20:15.643 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803812' 2024-05-08T17:20:15.643 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T17:20:15.643 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-05-08T17:20:15.795 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=60129542168 2024-05-08T17:20:15.795 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 60129542168 2024-05-08T17:20:15.795 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803812 1-60129542168' 2024-05-08T17:20:15.795 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T17:20:15.796 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-05-08T17:20:15.949 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=214748364803 2024-05-08T17:20:15.949 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 214748364803 2024-05-08T17:20:15.949 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803812 1-60129542168 2-214748364803' 2024-05-08T17:20:15.949 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2201: flush_pg_stats: for osd in $ids 2024-05-08T17:20:15.949 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-05-08T17:20:16.103 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2202: flush_pg_stats: seq=120259084298 2024-05-08T17:20:16.103 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2203: flush_pg_stats: test -z 120259084298 2024-05-08T17:20:16.103 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: seqs=' 0-25769803812 1-60129542168 2-214748364803 3-120259084298' 2024-05-08T17:20:16.103 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-08T17:20:16.103 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 0-25769803812 2024-05-08T17:20:16.104 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-08T17:20:16.105 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=0 2024-05-08T17:20:16.106 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 0-25769803812 2024-05-08T17:20:16.106 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-08T17:20:16.107 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=25769803812 2024-05-08T17:20:16.107 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.0 seq 25769803812' 2024-05-08T17:20:16.107 INFO:tasks.workunit.client.0.smithi172.stdout:waiting osd.0 seq 25769803812 2024-05-08T17:20:16.107 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 0 2024-05-08T17:20:16.556 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 25769803812 -lt 25769803812 2024-05-08T17:20:16.556 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-08T17:20:16.557 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 1-60129542168 2024-05-08T17:20:16.557 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-08T17:20:16.558 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=1 2024-05-08T17:20:16.559 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 1-60129542168 2024-05-08T17:20:16.559 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-08T17:20:16.561 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=60129542168 2024-05-08T17:20:16.561 INFO:tasks.workunit.client.0.smithi172.stdout:waiting osd.1 seq 60129542168 2024-05-08T17:20:16.562 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.1 seq 60129542168' 2024-05-08T17:20:16.562 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 1 2024-05-08T17:20:16.982 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 60129542168 -lt 60129542168 2024-05-08T17:20:16.983 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-08T17:20:16.983 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 2-214748364803 2024-05-08T17:20:16.983 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-08T17:20:16.985 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=2 2024-05-08T17:20:16.986 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 2-214748364803 2024-05-08T17:20:16.986 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-08T17:20:16.987 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=214748364803 2024-05-08T17:20:16.987 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.2 seq 214748364803' 2024-05-08T17:20:16.987 INFO:tasks.workunit.client.0.smithi172.stdout:waiting osd.2 seq 214748364803 2024-05-08T17:20:16.988 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-05-08T17:20:17.415 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 214748364802 -lt 214748364803 2024-05-08T17:20:17.415 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: sleep 1 2024-05-08T17:20:18.418 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: '[' 300 -eq 0 ']' 2024-05-08T17:20:18.418 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 2 2024-05-08T17:20:18.818 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 214748364803 -lt 214748364803 2024-05-08T17:20:18.818 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: for s in $seqs 2024-05-08T17:20:18.819 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: echo 3-120259084298 2024-05-08T17:20:18.819 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: cut -d - -f 1 2024-05-08T17:20:18.820 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: osd=3 2024-05-08T17:20:18.821 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: echo 3-120259084298 2024-05-08T17:20:18.821 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: cut -d - -f 2 2024-05-08T17:20:18.822 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2212: flush_pg_stats: seq=120259084298 2024-05-08T17:20:18.823 INFO:tasks.workunit.client.0.smithi172.stdout:waiting osd.3 seq 120259084298 2024-05-08T17:20:18.823 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: echo 'waiting osd.3 seq 120259084298' 2024-05-08T17:20:18.823 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: ceph osd last-stat-seq 3 2024-05-08T17:20:19.241 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: test 120259084298 -lt 120259084298 2024-05-08T17:20:19.242 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-05-08T17:20:19.242 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-05-08T17:20:19.242 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-05-08T17:20:19.808 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 5 == 0 2024-05-08T17:20:19.809 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-05-08T17:20:19.809 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-05-08T17:20:19.810 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-05-08T17:20:19.810 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-05-08T17:20:19.810 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-05-08T17:20:19.810 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-05-08T17:20:19.810 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-05-08T17:20:20.199 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-05-08T17:20:20.200 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-05-08T17:20:20.200 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-05-08T17:20:20.200 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-05-08T17:20:20.714 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 5 2024-05-08T17:20:20.715 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-05-08T17:20:20.715 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-05-08T17:20:20.715 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:354: TEST_rados_get_with_subreadall_eio_shard_1: delete_erasure_coded_pool pool-jerasure 2024-05-08T17:20:20.715 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:84: delete_erasure_coded_pool: local poolname=pool-jerasure 2024-05-08T17:20:20.715 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:85: delete_erasure_coded_pool: ceph osd pool delete pool-jerasure pool-jerasure --yes-i-really-really-mean-it 2024-05-08T17:20:21.242 INFO:tasks.workunit.client.0.smithi172.stderr:pool 'pool-jerasure' does not exist 2024-05-08T17:20:21.258 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:86: delete_erasure_coded_pool: ceph osd erasure-code-profile rm myprofile 2024-05-08T17:20:21.985 INFO:tasks.workunit.client.0.smithi172.stderr:erasure-code-profile myprofile does not exist 2024-05-08T17:20:22.002 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/erasure-code/test-erasure-eio.sh:42: run: teardown td/test-erasure-eio 2024-05-08T17:20:22.002 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-eio 2024-05-08T17:20:22.002 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-05-08T17:20:22.002 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-eio KILL 2024-05-08T17:20:22.003 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-08T17:20:22.003 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-08T17:20:22.004 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-08T17:20:22.004 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-08T17:20:22.004 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-08T17:20:22.316 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-08T17:20:22.317 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-05-08T17:20:22.318 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-05-08T17:20:22.318 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-05-08T17:20:22.319 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-05-08T17:20:22.320 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-05-08T17:20:22.321 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-05-08T17:20:22.322 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-08T17:20:22.322 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-05-08T17:20:22.322 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-05-08T17:20:22.323 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-08T17:20:22.324 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-05-08T17:20:22.325 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-05-08T17:20:22.325 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-eio 2024-05-08T17:20:22.352 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-05-08T17:20:22.352 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T17:20:22.352 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81110 2024-05-08T17:20:22.352 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.81110 2024-05-08T17:20:22.354 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-05-08T17:20:22.354 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-05-08T17:20:22.354 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2314: main: code=0 2024-05-08T17:20:22.354 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2318: main: teardown td/test-erasure-eio 0 2024-05-08T17:20:22.354 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-erasure-eio 2024-05-08T17:20:22.354 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2024-05-08T17:20:22.354 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-erasure-eio KILL 2024-05-08T17:20:22.355 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-05-08T17:20:22.355 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-05-08T17:20:22.355 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-05-08T17:20:22.355 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-05-08T17:20:22.355 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-05-08T17:20:22.358 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-05-08T17:20:22.358 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-05-08T17:20:22.360 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-05-08T17:20:22.360 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-05-08T17:20:22.361 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-05-08T17:20:22.361 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-05-08T17:20:22.362 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-05-08T17:20:22.362 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-08T17:20:22.363 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-05-08T17:20:22.363 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-05-08T17:20:22.363 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-08T17:20:22.364 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-05-08T17:20:22.366 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2024-05-08T17:20:22.366 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-erasure-eio 2024-05-08T17:20:22.367 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-05-08T17:20:22.367 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-05-08T17:20:22.368 INFO:tasks.workunit.client.0.smithi172.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.81110 2024-05-08T17:20:22.368 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.81110 2024-05-08T17:20:22.368 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-05-08T17:20:22.369 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-05-08T17:20:22.369 INFO:tasks.workunit.client.0.smithi172.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2319: main: return 0 2024-05-08T17:20:22.370 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-05-08T17:20:22.370 DEBUG:teuthology.orchestra.run.smithi172:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-05-08T17:20:22.424 INFO:tasks.workunit:Stopping ['erasure-code'] on client.0... 2024-05-08T17:20:22.424 DEBUG:teuthology.orchestra.run.smithi172:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-05-08T17:20:22.746 DEBUG:teuthology.parallel:result is None 2024-05-08T17:20:22.747 DEBUG:teuthology.orchestra.run.smithi172:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0 2024-05-08T17:20:22.758 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.0/client.0 2024-05-08T17:20:22.758 DEBUG:teuthology.orchestra.run.smithi172:> rmdir -- /home/ubuntu/cephtest/mnt.0 2024-05-08T17:20:22.816 INFO:tasks.workunit:Deleted artificial mount point /home/ubuntu/cephtest/mnt.0/client.0 2024-05-08T17:20:22.816 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-05-08T17:20:22.829 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-05-08T17:20:22.829 DEBUG:teuthology.orchestra.run.smithi172:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-05-08T17:20:22.896 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-05-08T17:20:22.907 INFO:teuthology.task.clock:Checking final clock skew... 2024-05-08T17:20:22.907 DEBUG:teuthology.orchestra.run.smithi172:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-05-08T17:20:22.927 INFO:teuthology.orchestra.run.smithi172.stdout: remote refid st t when poll reach delay offset jitter 2024-05-08T17:20:22.927 INFO:teuthology.orchestra.run.smithi172.stdout:============================================================================== 2024-05-08T17:20:22.927 INFO:teuthology.orchestra.run.smithi172.stdout:+hv01.front.sepi 67.205.162.81 3 u 73 128 377 0.181 -2.095 0.497 2024-05-08T17:20:22.927 INFO:teuthology.orchestra.run.smithi172.stdout:+hv02.front.sepi 63.231.80.2 3 u 17 128 377 0.172 -4.972 0.737 2024-05-08T17:20:22.927 INFO:teuthology.orchestra.run.smithi172.stdout:*hv03.front.sepi 74.6.168.72 3 u 55 128 377 0.152 -4.601 0.418 2024-05-08T17:20:22.927 INFO:teuthology.orchestra.run.smithi172.stdout: hv04.front.sepi .INIT. 16 u - 1024 0 0.000 +0.000 0.000 2024-05-08T17:20:22.928 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-05-08T17:20:22.939 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-05-08T17:20:22.939 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-05-08T17:20:22.965 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-05-08T17:20:22.999 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-05-08T17:20:23.041 INFO:teuthology.task.internal:Duration was 5666.583712 seconds 2024-05-08T17:20:23.041 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-05-08T17:20:23.066 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-05-08T17:20:23.066 DEBUG:teuthology.orchestra.run.smithi172:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-05-08T17:20:23.107 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-05-08T17:20:23.107 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi172.front.sepia.ceph.com 2024-05-08T17:20:23.107 DEBUG:teuthology.orchestra.run.smithi172:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-05-08T17:20:23.158 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-05-08T17:20:23.158 DEBUG:teuthology.orchestra.run.smithi172:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-05-08T17:20:23.270 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-05-08T17:20:23.270 DEBUG:teuthology.orchestra.run.smithi172:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-05-08T17:20:23.326 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-05-08T17:20:23.341 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-05-08T17:20:23.342 DEBUG:teuthology.orchestra.run.smithi172:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-05-08T17:20:23.353 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-05-08T17:20:23.390 DEBUG:teuthology.orchestra.run.smithi172:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-05-08T17:20:23.404 INFO:teuthology.orchestra.run.smithi172.stdout:kernel.core_pattern = core 2024-05-08T17:20:23.414 DEBUG:teuthology.orchestra.run.smithi172:> test -e /home/ubuntu/cephtest/archive/coredump 2024-05-08T17:20:23.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-08T17:20:23.461 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-05-08T17:20:23.477 INFO:teuthology.task.internal:Transferring archived files... 2024-05-08T17:20:23.479 DEBUG:teuthology.misc:Transferring archived files from smithi172:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/lflores-2024-05-08_14:59:36-rados-wip-lflores-testing-2-2024-05-07-1606-squid-distro-default-smithi/7697621/remote/smithi172 2024-05-08T17:20:23.479 DEBUG:teuthology.orchestra.run.smithi172:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-05-08T17:20:23.525 INFO:teuthology.task.internal:Removing archive directory... 2024-05-08T17:20:23.525 DEBUG:teuthology.orchestra.run.smithi172:> rm -rf -- /home/ubuntu/cephtest/archive 2024-05-08T17:20:23.569 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-05-08T17:20:23.582 INFO:teuthology.task.internal:Not uploading archives. 2024-05-08T17:20:23.582 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-05-08T17:20:23.609 INFO:teuthology.task.internal:Tidying up after the test... 2024-05-08T17:20:23.610 DEBUG:teuthology.orchestra.run.smithi172:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-05-08T17:20:23.617 INFO:teuthology.orchestra.run.smithi172.stdout: 658614 4 drwxr-xr-x 2 ubuntu ubuntu 4096 May 8 17:20 /home/ubuntu/cephtest 2024-05-08T17:20:23.618 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-05-08T17:20:23.641 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-05-08T17:20:23.711 INFO:teuthology.run:Summary data: description: rados/standalone/{supported-random-distro$/{ubuntu_latest} workloads/erasure-code} duration: 5666.583711862564 flavor: default owner: scheduled_lflores@teuthology success: true 2024-05-08T17:20:23.711 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-05-08T17:20:23.793 INFO:teuthology.run:pass